Grids

Need to be able to summarise points a lot more.
This could come when I understand the domain to a better degree - for now I need to maintain a narrative in the points as an anchor for myself.

There seems to be emerging a picture of designers stepping out of the safe confines and daring to dirty themselves in "messy" situations even if they get flak for it from their more traditional, "scientific" colleagues.

Maybe it is not even a conscious decision to go against the flow.
Maybe the naturalistic setting in which learning occurs, with so many individuals, with so many variables could just no longer be ignored?
Is it going to be possible to fit this into the traditional theoretical setting?
How much does teaching have to change?
How much does traditional, scientific, theory have to change?
Surely if Nobel Laureates have stepped away from certain traditions in order to enable them to make the quantum leaps they have this says something about the confines of the tradition and how it may no longer fit with enabling new discoveries?
BUT!
First they got their scientific, theoretical grounding.
Then this enabled them to critically evaluate it to see how it did not fit to with allowing them jump to see things from a completely different paradigm.

I wish there was some group of peers I could debate this with.

Attempt at grid to Manage Literature

Author	What is DBR	How is DBR done/Suggestions	Limitations	Advantages	Elements inherent to the design that differ….
Kelly 2004	-Emerging, -Loose set of methods, -Produces an artefact (concrete or conceptual) through an iterative process, that outlasts the study and is adopted, adapted, used by others. -Object of study – the process of engagement between the teacher and the student (where researcher could also be teacher). -The design experiment = teaching. -One end of spectrum - Centrally concerned with curricular subject matter (Is this QA?maybe include that instead of VS) -Other end of spectrum – produces a software artifact that contains the model of learning or teaching (RC?) not for the software's sake but as a tool to explore and test research questions about learning. -occur in real-world settings - raw material of design studies inescapably contingent, often unpredictable and perhaps unrepeatable	- stage appropriate use of research methods	-Focus of attention of design studies has not been on its later adoption - Needs to move from being a loose set of methods to being a methodology GRAMMAR- lacks argumentative grammar that supplies the logic of the methodology with regards the reasoning about its data DEMARCATION- must clarify how results contribute to demarcation (sound and unsound claims) and/or problems of meaningfulness ACTORS- without experimental control can there be generalizations? (QA?) - are causal attributions plausible when there are multiple dependant variables - where the product is ephemeral (learning) in educational settings and the status of claims is meant to be general does flexible design revision make sense? - should not the design also forefront indiviudual cognition and link to other areas of science - with regards coparticipant involvment during research what impact does this involvment have on the inferential character of any finds - whose voice takes precedence and why? Many studies have small sample size so cannot be generalized over N (RC and QA large sample size VS small) CONTEXTS - targeted perturbations weakens generalizations to cognitions in larger and normal settings (This is Can look for) lack of structural or statistical controls/ untested alternative hypotheses - no guidelines to minimize researcher bias	-generate hypothesis and frameworks -contribute to model formulation vs model estimation or validation - consruct advancing hypothese could emerge from studies -is a means of going beyond a literature review that does not provide a good question or inhibits progress because ot the models guiding it. - can contribute to the problem of meaningfulness, that precedes and is the foundation for addressing problems of demarcation contribute to generalizations as they contribute to the thinking of other researchers in other domains – may guide observations, suggest variables, help make sense of data
Reimann

Why are there so many references to DBR not being accepted?

(Kelly 2004) asks is Design Studies a loose set of methods or a methodology?

Design produces an artifact that outlasts the study and can be "adopted, adapted and used by others"

The object of study is the "process of engagement" between the student and the teacher.

Experiment = testing of hypotheses and conjectures

Design of software involves engineering a broader learning environment

Questions about learning are incorporated into the software – reified, explored, tested and this allows for its use and testing elsewhere

Software (or conceptual framework) can then be used by others – Roschelle 2000 says without it one may not even be able to evoke cognitive learning in the students – interesting concept

He argues that the set of process descriptors that are used to define DR do not enable DR, as yet, to be defined as a method, as they do not define the conceptual strucure and do not compley with the definition of a method namely "a procedure, a process, a set of steps to follow.

This is required in order to be able to meet the claims it makes such as enabling one to "elicit a generative frame- work, advance our understanding of disciplinary knowledge, serve as an incubator for new research techniques, advance our skill in designing learning environments or lead to better instrumentation"

According to Kelly a mature methodology has at least some of following characteristics:

ARGUMENTATIVE GRAMMER

The logic that supports the use of the method and the reasoning about its data (supplies the logos in the methodology). It can be viewed separately from the examplar.

Some questions that can be asked to determine it are:

"What guides the reasoning with these data to make a plausible argument?"

"What is the separable structure that justifies collecting certain data and not others and under what conditions?"

Reviewers may not reject the studies based on the choice of method but due to the violaton of the logos that one would expect to see with that choice of method.

CONTRIBUTIONS TO PROBLEMS OF DEMARCATION OR MEANINGFULNESS

The problem of demarcation is the application of argumentative techniques that distinguish scientific claims from pseudoscience or methaphysical claims (example given is how scientific arguments backed by empirical methods can support astronomy and debunk astrology)

Kelly looked at Collins' description of design studies and asked the following questions that tried to isolate the demarcation of such studies:

-   regardless of the 'richness of descriptions' if there are not experimental controls how can one generalize to other settings (QA – no experimental control, but RC yes)

-   where multiple dependant variables are present (and not controlled), how does one  breakdown the complex interactions in order to determine causal attributions?

-   Where the 'product' is ephemeral ie learning as opposed to an engineering product, and the claims are general does allowing for flexible design revision make sense? But as discussed earlier with regards the software as an artefact – is learning the actual product? An engineered product for transport – a car, the product is the car and WHAT IS THE TRANSPORT? In DR the product is the software or conceptual framework and the transport is learning.???? Look into this further.

-   Is something forgone when the social interaction is valued over social isolation (Collins), does this map to foregoing focus on individual cognition (as Kelly sees it) and can findings thus not be linked it to other areas of science?

-   Kelly asks what is the basis for developing profiles of the learner along a number of dimensions and not others

-   He questions the meaning of hypothesis testing in situations that are fluid, messy (I don’t really understand the grounds for the issue here ??? Look into this further – is it because one cannot keep all other things equal and isolate the other causal effects to isolate the causes of one's hypotheses?)

-    

GENERALIZATIONS OVER ACTORS

As studies focus on a few subjects there is a problem of generalization to a larger N and it weakens the methodology as it lacks sampling and descriptive power.

In addition because the cognitive responses of targeted students are responses to targeted perturbation that are part of the design experiment, Kelly argues that the sampling prolem over actors is compounded as it weaknes the generalization to a larger student body and normal cognitions.

So is this term "generalizations over actors" a standard term with specific meaning in the domain??? (Look into this) and could this be the reason for many scaling up issues that are referred to in the various experiments?

GENERALIZATIONS OVER BEHAVIOURS

Causal claims about behaviour are weakened when there is little or no structural or statistical controls via an experimental control

GENERALIZATIONS OVER CONTEXT

Standard ethnography does not place intervention and iteration centrally – whereas the context in DR requires that the context be engineered, the context is a designed environment. So this could be a reason that the findings may not be generalized to other contexts?

Basic Design Ehtnography describes and interprets a particular culture.

Kelly then describes the "design ethnography" behind QA – which goes a step further to critique and change the social commitments of that culture. These changes and critique emerge as an artifact much as the learning artifact vs the product car in the previous paragraph.??? I need to ponder on this and to see if my understanding is correct.

Kelly points out that the emerging design ethnography methods need to be spelled out and its strengths and weaknesses shown. (Slight difference in how I am explaing this to how it appears in the article – Kelly states "explicitly spell out the methodological strengths and weakness of the emerging design ethnography methods")

PROBLEM OF MEANINGFULNESS

Kelly describes the different viewpoints that exist with regards to the validity of scientific discoveries emerging from pure scientific method vs eureka moments that are the result of processes of thought founded on the imaginative and inspirational. Medawar's quote gives credence to the process whereby the imaginative comes first and it is then followed by the scientific facts and acts.

Design studies could thus be seen as a step BEFORE models estimation and validation ie 'model formulation' ie they generate models and hypothesis

To back up the validity of this process, Kelly quotes, Russel Hulse (Nobel Laureate in physics, 1993) argument that any infinite number of hypothesis could be tested– testing hypotheses in the traditional scientific manner does not, by itself, create useful knowledge, rather (powerful) hypotheses could emerge from studies directed at the problem of meaningfulness and these can then be tested.

So, good hypotheses (questions) advance science, but how can they be generated?

From a literature review? Kelly asks how the question emerges and raises the point that sometimes a literature review actually inhibits progress and quotes the nobel laureate Müller to back this up. Kelly suggests that constructs that are found in literature reviews have emerged from design contexts and it is these studies that may "promote the identification and growth of new ides and constructs.

So does this mean that design studies are placed in a different position on the continuum of scientific methodology? Instead of seeing them as hypothesis testing are they hypothesis generating AND through their iterative processes become hypothesis testing as they advance within their own paradigm. Is this unique to them? Is this an area in which they can be differentiated??? I need to look into this further

GENERALIZATIONS OF CONCEPTUAL FRAMEWORKS OR ARTICUALATION STRATEGIES

Kelly appears to see Design Studies as contributing to ideas for further research. Conceptual frameworks in design studies emerge from the authentic settings and experiences of participants and thus guide further observations, delimit variables to study, contribute to sense-making of the data. Later studies with methodologies that adhere more to "scientific methods" can then extend this to determine generalizations across actors, behaviour and context.

So does this mean that trying to position Design Studies into a Scientific Mould is negating an aspect of it ???? I need to understand this better. Could design studies be split into two different components one hypothesis generating and one hypothesis testing. The hypothesis testing component should adhere to a specified methodology but the criteria the scientific community use to assess the hypothesis generating component and how designers go about this should use another lens to view it? Should I look at VS and RC with a view to differentiating these 2 components?

RESTRICTIONS ON RESEARCHER BIAS

Mature methodologies have guidelines for minimizing the occurrence of inevitable researcher bias.

I need to have a better understanding of the domain to grasp what Kelly is referring to in this section.

Kelly suggests that 'PART OF THE METHODOLOGICAL WORK FOR DESIGN STUDIES IS TO CLARIFY IN WHICH STAGE AND FOR WHAT PURPOSES METHODS ARE APPROPRIATE AND INAPPROPRIATE"

BALANCING CONTINGENT WITH NECESSARY CLAIMS

Contingent = arbitrary vs Necessary

Kelly points out that as Design Studies occur in naturalistic settings the raw materials are contingent, upredicatable and unrepeatable.

Outputs are going to be considered simply descriptive by others until the components can be modelled on what is necessary and thus be considered more scientific. His opinion is that when the contingent is connected to the necessary that is when theory-building occurs.

He refers to 3 examples in McCandliss et al. which I might have to refer to.

In addition he suggests it is necessary to collaborate with other methodologists to develop cross-disciplinary initiatives to the behavioural, cognitive, social sciences and the example of the cognitive neurosciences in order to develop aspects of model building or later stages of research-as-design. For this he makes 2 references Sloane & Gorard and Bannan-Ritland which I am going to consult to see what he means.

PRODUCING USABLE KNOWLEDGE.

Kelly states that Educational researchers have a difficult task in that they need to; answer to a variety of audiences, their claims are difficult to substantiate and they have to include these claims in curriculum to fit into the classrooms.

Educational Research acts on and perturbs the systems in which they work and this adds complexity to the scientific problem. He states that "the laboratory scientist's 'error variance' is the educational researcher's reality.".

Kelly thus suggests that researchers in this paradigm have to exceed the criteria for scientific claims. He asks whether this research can produce learning/teaching artefacts that are efficient, workable and economical and do not require a high-cost switch from current practises. Designers must include in their design (evaluative criteria) factors that will cater with the problems inherent in the later adoption and adaptation.

SUMMARY

It appears that Kelly is saying that Design Research is not accepted as a scientific method but that the reasons why this is so are the reasons inherent in what this Research designs (learning outcomes) and the contexts in which they occur (messy, naturalistic settings). He seems to suggest that by working with other disciplines solutions for these issues could be found. Another factor that could lead to this design being more scientific is by breaking it up into different stages and looking at what is appropriate in these stages (for this he refers to the Zaritsky reading).

To be Honest

DBR -methodological approaches- quantitative data - yeah, yeah I just slip into a coma.
However,
Google Scholar Search on The role Ethnology plays in DBR....
and then get

Issues of Validity in Openly Ideological

Research: Between a Rock and a Soft Place

that opens with 

The attempt to produce value-neutral social science is increasingly being aban-doned as at best unrealizable, and at worst self-deceptive, and is being replaced by social sciences based on explicit ideologies. Mary Hesse (1980)

Now that warms my cockles, that entices me. Can I make something of it tomorrow??

Coupled with Barab maybe?

Another one that doesn't put me to sleep is

Collins, Allan , Joseph, Diana and Bielaczyc, Katerine(2004) 'Design Research: Theoretical and Methodological Issues', Journal of the Learning Sciences, 13: 1, 15 — 42

Tomorrow the birds will wake me at the crack of dawn again and hopefully I will be able to formulate it.....

Don't know how I am going to get out of this one

(Fishman, Marx et al. 2004) DBR does not explicitly address systemic issues of usability, scalability and sustainability.

This could be a research avenue – as been suggested – and sticking to articles from this issue of the JLS – BUT

This was in 2004????? View the pace of things surely the picture looks very different now?

If I stick to this issue am I stuck in the past (as my kids assure me I am)?

If I don't stick to this issue I am off on my tangent again – a never ending literary review down the yellow brick road!!!!!!!!!

Fishman, B., R. W. Marx, et al. (2004). "Creating a Framework for Research on Systemic Technology Innovations." Journal of the Learning Sciences 13(1): 43-76.

           

The problem of the question again!

As I struggle to conceptualise Assignment 2 – I took a step back and tried to find a copy of a "Research Design Report" using Google Scholar – I did find this reference –

Leedy, P. D. and J. E. Ormrod (2001). The Problem: The heart of the Research Process. Practical research: planning and design. Upper Saddle River, N.J, Merrill Prentice Hall.

           

I am seeing the change with regards books now. It is too time consuming to actually call up a refernce for a book, go to the library to get it and scan (read) all the pages manually. I want to call up an electronic version, immediately, and be able to use my PDF annotation tool.

On Sydney Library there was only chapter 3 available electronically and not in .pdf format – ok, maybe good enough.

This Chapter informs us that the first step is to Define the Research Problem Question.

The question should be:

1. Important – the answer should make a difference
2. It should advance – the frontiers of knowledge

I had thought that my Research Problem could be the fact that I did not have a clear enough understanding of DBR and research could advance MY frontiers of knowledge–

WHAM

First page the authors knocked that out of the window

1. Research projects should not be a ruse for achieving self-enlightenment

They suggest

Read the literature: to find out what is already known and what needs to be.

Attend Professional conferences

Seek the advice of experts.

The experts have suggested I do a study of DBR – but with the level of my understanding and knowledge of the subject matter at the moment – any topic I chose I would feel that I am being presumptious!!!

The literature is vast!!!  The debate and research has been gong on for years. I have to immerse myself in following the evolution to be secure enough in my knowledge to presume the ability to highlight issues that need to be further researched???

The other students can so it must be a problem in me???

Leedy suggests the following steps to clarify once a problem question has been found.

1. State the problem clearly and completely – (I will have to come back to this one)
2. Think through the feasibility of the project that the problem implies ( time issue once again rears its head with the Masters!!!!- okay so it is not a COMPLETE research project) – look at limits and focus!
3. Say precisely what you mean (I have a problem with doing this I must be clearer in my mind to be able to formulate things clearly)
4. Edit your words – to converge on a clear, conscise problem statement – "sharpening a thought to a gemlike point"

Define subproblems vs pseudosubproblems:

I should be aiming at the setting of the problem:

1. Stating the hypothesis and or / research question

- this will provide guidance for the kids of the data that should be collected and how to go about analyzing and interpreting them. Normally a one to one mapping between questions and subproblems. Cant set about trying to prove an hypothesis – it can only be supported or not by the data

2. Delimiting the research

and stick to these limits, no matter how enticing or interesting other aspects may be

3. Defining the terms

each term must be defined operatively

a formal definition contains 3 parts a) the term to be defined b) the genera, general class to which the concept being defined belongs and c) the differentia, the specific characteristics/traits that distinguish it from all other members of the general classification.

Avoid circular definitions

5. Stating the assumptions

All assumptions that have a material bearing on the problem should be explicitly stated.

To determine ones assumptions ask "What am I taking for granted with respect to the problem"

General formats of Research proposals

1. First Chapter

Present the problem and its setting

Hypothesis or questions concerning the subproblems are stated

The settings of the problem are presented – statement of the delimitations, definitions of the terms, assumptions , the importance of the study.

2. Review of the Related Literature

A discussion of investigations that others have done.

The chapter provides a checklist to evaluate the draft proposal pg 66-67.

For fine tuning:

Complete the necessary background search – do I know enough about the topic that I can ask important questions?

Try to see the problem from all sides – what is good/not about the potential project

Think through the process – from the lit review, through implementation, data collection, data analysis, interpretation. Pay close attention to bottlenecks and pitfalss that could cause problems later on.

Use all available tools and resources at your diposal – allow time for larning about new tools or how to use old tool sin new ways – NO KIDDING!!! (.pdf annotations, googledocs, endnotes……

Discuss your research problems with others, especially peers – pity we don't really have that set up. People are distanced and so stressed to get their own done.

Hold up your project for other to examine and comment on – have to rely on supervisors for that and Lina always does a great job.

Actively seek information and constructive criticism that may help you accomplish your task

Remember that your project will take time -  I never forget this