Category Archives: survey

Research update #61: The massive data dump

I have a reason for not having posted for a little while, for a change. I’ve been swimming in the data from my first survey, learning about statistics and stats tools and generally working to get my head around what the survey tells me about the edvisor landscape, how our work is perceived and valued and what activities/knowledge might be connected more to one of the three roles than others.

I’m in the process of putting together a document capturing the many facets of this data – some points stand out more than others – and it is currently around 192 pages. This includes a LOT of bar charts and I’m still pulling everything together to be able to go through and try my hand at analysing things down to a few pithy pages for the thesis.

Meanwhile, I had a slot in the most recent TELedvisors webinar, among some people that I’d like to consider peers also conducting research in this space – Evonne Irwin (Uni of Newcastle), Natalia Veles (James Cook University) and Karin Barac (Griffith Uni). (To be honest though, their presentations all meshed together theory and practice so cleverly that peer feels slightly aspirational)

I think that because I have spent so much time in recent months with this data, and so little time discussing it with anyone, I got a bit caught up and turned my 10 mins into something of a data dump. The information was there but not so much the discussion about why it mattered. Something to consider more next time.

Anyway, for what it’s worth, here’s what I had to say.

Thoughts on: Project Management in Instructional Design (Allen, 2020)

Reading this thesis was valuable because it showed how stark the difference can be between them. Where Amparo took a very qualitative, narrative driven approach to the research going narrow but deep with 3 people and case studies, Allen goes shallow but broad with this quantitative research into the most valued project management competencies for instructional designers (IDs).

Allen takes a deep dive into a comparatively rich pool of literature relating to the project management skills that best serve IDs, generating a comprehensive literature review that sums up the last decade or so of research in this space very effectively. She builds on it by conducting her own two stage survey of 86 IDs in a range of sectors (Higher Ed, corporate, ID project team leaders) to gather some rich quant data.

Nothing overly surprising emerges in terms of the favoured competencies, with the data largely aligning with the studies that had come before (but at a larger scale) but it did spark a few thoughts for me about my own work. Some of the competencies across the literature I personally found a little nebulous – things like “attention to detail” which are certainly valuable professional attributes but, coming from a competency based education background, I was curious about how that might be meaningfully measured or taught. This brought me back to realising that I need to think carefully about what practices, attributes and competencies mean in the data I am gathering.

The painstaking detail in the writing about the work undertaken, from the lit review to the data collection and analysis offered a useful benchmark for my own future writing.

It was also useful to scan the references and find a few promising leads that I’ve previously missed. These include:

Kenny, J. (2004). A study of educational technology project management in Australian universities. Australasian Journal of Educational Technology, 20(3), 388–404.

Allen, M. (1996). A profile of instructional designers in Australia. Distance Education 17(1), 7–32.

I don’t think I had thought to search for IDs in the Australian literature.

Allen, S. A. (2020). PROJECT MANAGEMENT IN INSTRUCTIONAL DESIGN. Franklin University.

Week #4 SOCRMx – Reflecting on methods

This week in the Social Research Methods MOOC we take a moment to take a breath and consider the approaches that we currently favour.

One of the activities is to reflect in our blog – so I guess this is that. I’m looking at surveys because I still need to get my head around discourse analysis, not having really used it before.

Reflecting on your chosen methods

Choose one of the approaches you’ve explored in previous weeks, and write a reflective post in your blog that answers the following questions. Work though these questions systematically, and try to write a paragraph or two for each:

What three (good) research questions could be answered using this approach?

I’m fairly focused on my current research questions at the moment and I would say that using surveys will help me to start answering them, but I certainly wouldn’t rely solely on surveys. The questions are: How do education advisors see their role and value in Tertiary Education? How are education advisor roles understood and valued by teachers and institutional management? What strategies are used in tertiary education to promote understanding of the roles of education advisors among teaching staff and more broadly within the institution.

What assumptions about the nature of knowledge (epistemology) seem to be associated with this approach?

The main assumption is that subjective or experience based knowledge is sufficient. I don’t believe that this is the case. Clearly, a survey can be useful in collecting broad data about the attitudes that people claim or even believe that they hold however people can have a tendency to want to see themselves in the best possible light – the heroes of their own story – and responses might be more indicative of what people would like to think they believe than what their actions show them to believe.

What kinds of ethical issues arise?

This would depend on the design of the research. Assuming there is no need for participants to be subsequently identifiable, anonymity should enable respondents to express their opinions freely and without concern for consequences. Questions should be designed in a way that is not unnecessarily intrusive or likely to influence the way that respondents answer. I’d also assume that good research design would ensure that the demographics of survey participants is reflective of that community.

What would “validity” imply in a project that used this approach?

I would say that ‘validity’ would require addressing some of the issues that I’ve already raised. Primarily that the survey itself could be relied upon to collect data that accurately reflects the opinions of the survey respondents without influencing these opinions or asking ambiguous questions that could be interpreted in different ways. My overall preference would be for the survey to be one part of a larger research project that provides data from different sources that can be used to provide greater ‘validity’.

What are some of the practical or ethical issues that would need to be considered?

The survey would need to be anonymous and the data kept securely. Questions should be designed to be as clear and neutral as possible and a sufficiently representative sample of participants obtained. Given the number of surveys that people get asked to complete these days, ensuring that people have a clear understanding of the purpose and value of the research would be vital. For the same reason, I’d suggest that we have a responsibility to ask people only for the information that we need and nothing more.

And finally, find and reference at least two published articles that have used this approach (aside from the examples given in this course). Make some notes about how the approach is described and used in each paper, linking to your reflections above.

Mcinnis, C. (1998). Academics and Professional Administrators in Australian Universities: dissolving boundaries and new tensions. Journal of Higher Education Policy and Management, 20(2), 161–173.

Comparison of two surveys, one of academic staff (1993) and one of administrative/professional staff (1996). Analysis of results, some additional questions were added to the second survey

Wohlmuther, S. (2008). “Sleeping with the enemy”: how far are you prepared to go to make a difference? A look at the divide between academic and allied staff. Journal of Higher Education Policy and Management, 30(4), 325–337.

Based on an anonymous online survey of 29% of all staff – academic and professional at her institution, which included questions about demographics, perceptions of the nature of their roles, the ‘divide’ and the value of different types of staff in relation to strategic priorities.

Both surveys related to workplace issues and attitudes, which meant that privacy was a significant factor. I was less impressed with the approach taken by Wohlmuther, which I felt was overly ambiguous in parts.

“Survey respondents were asked what percentage of their time they spent on allied work and what percentage of their time they should spend on allied work. The term ‘allied work’ was not defined. It was left to the respondent to interpret what they meant by allied work” (p.330)

I do still think that I’ll use surveys as a starting point but expect to then take this information and use it to help design interviews and also to inform analysis of other sources of data.

Kahoot! – handy online quiz activity builder supporting mobile access in class

https://create.kahoot.it/#

Kahoot is a very simple but highly interactive online tool that enables teachers to create quizzes (or survey questions) that students are able to access via mobile devices. It takes a fairly gamified approach with time limits for responses to questions, points for faster responses and leaderboards

via Delicious (via IFTTT)