Study Results
Aggregated responses and participation overview
v1

Study Funnel

20
Participant Sample
9No AI Group
11AI Group
Group Allocation
3
7
Solved Task 1
2
1
Solved Task 2

Filters

-1 excluded

Totals

Participants (All)
20
Completed Task 1
10
Completed Task 2
3
Task 1 Avg Correct (0-1)
0.56
Task 1 Respondents
20
Task 2 Avg Correct (0-1)
0.33
Task 2 Respondents
7
Task 1 Correct Answers (No AI Group vs AI Group)
No AI Group56%
AI Group56%
Task 2 Correct Answers (No AI Group vs AI Group)
No AI Group39%
AI Group29%

Participants

Participant P130501AI Group
Task 1 completedSoPra
23 yrs
BSc studentBSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
SoPra
AI in workflow
If I have an idea how to do things, then ask AI whether it is possible, how the syntax looks like
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 0 · Total 6/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Easy (2)
Mental effort
(4) Rather low mental effort
Confidence
Slightly confident (2)
What were the biggest difficulties you encountered when completing your task (if any)?
Understanding the connections between the different files, but once I had a mental graph of the connections I could get started with changes and also use ChatGPT to help me, with it doing all of the work in the end it feels like to me.
Task 2
Coding
Comprehension Quest.
Q1: 0 · Q2: 2 · Q3: 0 · Total 2/12
Subjective ReportingParticipant's subjective reporting after working on the second task
Task Difficulty
Difficult (4)
Mental effort
(6) Rather high mental effort
Confidence
Not at all confident (1)
What were the biggest difficulties you encountered when completing your task (if any)?
Creating the correct SQL statement which I probably did not succeed and is the reason why I did not finish. Also the problem how function calls seemingly did not work but I was just looking in the wrong place for console logs.
Participant P130502AI Group
Task 1 completedTask 2 completed
23 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
5-6 projects. Expert
AI in workflow
Generate documents and descriptions
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 0 · Q3: 4 · Total 6/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Easy (2)
Mental effort
(4) Rather low mental effort
Confidence
Extremely confident (5)
What were the biggest difficulties you encountered when completing your task (if any)?
Understanding the codebase as it is just a test code base a lot of stuffs are done in the way that it shouldn't be.
Task 2
Coding
Comprehension Quest.
Q1: 4 · Q2: 2 · Q3: 0 · Total 6/12
Subjective ReportingParticipant's subjective reporting after working on the second task
Task Difficulty
Very Easy (1)
Mental effort
(3) Low mental effort
Confidence
Extremely confident (5)
What were the biggest difficulties you encountered when completing your task (if any)?
N/A
Participant P160501No AI Group
Task 1 completedTask 2 completed
24 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
Working as a web developer for 2 years (40%) Bestande - student project for 1 year
AI in workflow
I use it mostly to generate code of things I don't know how to do (instead of Googling). I also use it to search for best practices if I'm not sure if I do something in the best way. If I know how to do something I don't use AI. I haven't used co-pilot for a year.
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 0 · Total 6/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Medium (3)
Mental effort
(7) High mental effort
Confidence
Very confident (4)
What were the biggest difficulties you encountered when completing your task (if any)?
That you need to add the response of the POST-request with res.json(registeredUser); · (instead of just return registeredUser;) · Getting the correct SQL Lite Insert statement
Task 2
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 0 · Total 6/12
Subjective ReportingParticipant's subjective reporting after working on the second task
Task Difficulty
Medium (3)
Mental effort
(6) Rather high mental effort
Confidence
Very confident (4)
What were the biggest difficulties you encountered when completing your task (if any)?
I didn't think that the functions from bcrypt to hash and compare a password returned a Promise which lead to a bug.
Participant P190501AI Group
Task 1 completed
21 yrs
BSc studentBSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Coding
Comprehension Quest.
Q1: 0 · Q2: 0 · Q3: 4 · Total 4/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Difficult (4)
Mental effort
(6) Rather high mental effort
Confidence
Slightly confident (2)
What were the biggest difficulties you encountered when completing your task (if any)?
correctly connecting front and backend because if I want the AI to fix it for me, it requires multiple files and functions. In my case, I spent a lot of time just realizing that my code has one unnecessary return null.
Task 2
Coding
Comprehension Quest.
· Total 0/0
Subjective ReportingParticipant's subjective reporting after working on the second task
Task Difficulty
Easy (2)
Mental effort
(4) Rather low mental effort
Confidence
Moderately confident (3)
What were the biggest difficulties you encountered when completing your task (if any)?
I wans't able to solve the problem of redirecting to logged-in page and it was hard to figure out what i needed to modify.
Participant P190502AI Group
Task 1 completedSoPra
20 yrs
BSc studentBSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
in SoPra only
AI in workflow
Use AI all the time
Task 1
Coding
Comprehension Quest.
Q1: 4 · Q2: 0 · Q3: 2 · Total 6/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Medium (3)
Mental effort
(5) Neither low nor high mental effort
Confidence
Slightly confident (2)
What were the biggest difficulties you encountered when completing your task (if any)?
At the end i still didnt really know what was going on. Therefore it was hard when bugs occurred.
Participant P190504No AI Group
26 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
I dont have.
AI in workflow
Frequently.
Task 1
Coding
Comprehension Quest.
Q1: 0 · Q2: 0 · Q3: 0 · Total 0/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Very Difficult (5)
Mental effort
(5) Neither low nor high mental effort
Confidence
Not at all confident (1)
What were the biggest difficulties you encountered when completing your task (if any)?
Lack of GPT. Without gpt, i couldnt decide where to start and i just lost my confidence.
Participant P210501No AI Group
SoPra
24 yrs
BSc studentBSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
Sopra Wordpress
AI in workflow
Fix errors and find bugs
Task 1
Coding
Comprehension Quest.
Q1: 0 · Q2: 4 · Q3: 4 · Total 8/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Difficult (4)
Mental effort
(8) Very high mental effort
Confidence
Not at all confident (1)
What were the biggest difficulties you encountered when completing your task (if any)?
socket hangup error(probably due to OpenDB). · For me it was difficult to find proper documentation for OpenDB implementation with react and the documentation on GitHub was not very helpful.
Participant P230501AI Group
SoPra
22 yrs
BSc studentBSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
Web development for SoPra, mainly Frontend using ReactJS
AI in workflow
Using Gemini to help explain code for LLMs for understanding and finding problems in code
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 4 · Total 10/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Difficult (4)
Mental effort
(8) Very high mental effort
Confidence
Moderately confident (3)
What were the biggest difficulties you encountered when completing your task (if any)?
I had difficulties understanding the necessity of the individual steps. First i had to understand the architecture and understand which component uses which other component and how they communicate. I found it difficult to know where exactly what is being defined. Because there is a User.ts file, the auth controllers file and also the internalClient. They all contain a registerUser functionality, so when the internalClient also had a registerUser method i was confused on whether i had correctly understood the task. With ChatGPT i was advancing and it helped me understand what was going on, but the problem then arose to find out why my code wasn't working and why i couldn't register. Trying to bugfix and actually find out what was being sent and why i couldn't register my user was hard, because i wasn't quite sure what i was doing.
Participant P230503AI Group
Task 1 completed
23 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
bachelor’s projects, pet projects, internship projects
AI in workflow
to learn, to understand bugs or errors I face
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 4 · Total 10/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Difficult (4)
Mental effort
(7) High mental effort
Confidence
Moderately confident (3)
What were the biggest difficulties you encountered when completing your task (if any)?
types and getting into the the project structure
Task 2
Coding
Comprehension Quest.
Q1: 4 · Q2: 0 · Q3: 0 · Total 4/12
Subjective ReportingParticipant's subjective reporting after working on the second task
Task Difficulty
Difficult (4)
Mental effort
(8) Very high mental effort
Confidence
Slightly confident (2)
Participant P230504No AI Group
22 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
AI in workflow
I use it as a study buddy
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 2 · Total 8/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Difficult (4)
Mental effort
(7) High mental effort
Confidence
Slightly confident (2)
What were the biggest difficulties you encountered when completing your task (if any)?
Fixing backend encountering error during the registration request 500
Participant P240501No AI Group
23 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
mockup demos, prototypes for course project Online shopping, Library management system, Hospital Information System
AI in workflow
Use it to name items, add comments, find the reasons for some small bugs
Task 1
Coding
Comprehension Quest.
Q1: 0 · Q2: 4 · Q3: 4 · Total 8/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Very Difficult (5)
Mental effort
(1) Very very low mental effort
Confidence
Not at all confident (1)
What were the biggest difficulties you encountered when completing your task (if any)?
I am very hard to find a example for the technology stack.
Participant P260501AI Group
Task 1 completedSoPra
28 yrs
BSc studentBSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
SoPra
AI in workflow
My Bachelor's Thesis is based on evaluating biases of GenAI.
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 0 · Q3: 0 · Total 2/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Medium (3)
Mental effort
(6) Rather high mental effort
Confidence
Moderately confident (3)
What were the biggest difficulties you encountered when completing your task (if any)?
To see the big picture while solving the task.
Task 2
Coding
Comprehension Quest.
· Total 0/0
Subjective ReportingParticipant's subjective reporting after working on the second task
Task Difficulty
Very Difficult (5)
Mental effort
(8) Very high mental effort
Confidence
Not at all confident (1)
What were the biggest difficulties you encountered when completing your task (if any)?
I do not understand how db requests work and how those different files are connected with each other/how they need to work with each other.
Participant P260502AI Group
SoPra
24 yrs
BSc studentBSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
Software Engineering Lab
AI in workflow
I use it to find errors, tell me how given code works and on notation of code (e.g. TypeScript notation)
Task 1
Coding
Comprehension Quest.
Q1: 0 · Q2: 4 · Q3: 2 · Total 6/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Difficult (4)
Mental effort
(7) High mental effort
Confidence
Not at all confident (1)
What were the biggest difficulties you encountered when completing your task (if any)?
I have no previous experience in larger code bases that use this exact tech stack (SoPra was the only one, but the tech stack was different), so it took me a while to get used to what all components are doing and how they interconnect. E.g. I have never used GraphQL or JWT token.
Participant P260503AI Group
SoPra
25 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
Sopra, advances software engineering modules at uzh
AI in workflow
I use it to help me code an ios app with swift
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 4 · Total 10/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Difficult (4)
Mental effort
(7) High mental effort
Confidence
Moderately confident (3)
What were the biggest difficulties you encountered when completing your task (if any)?
I never worked with express before, so I did not how it worked.
Participant P260504No AI Group
SoPra
22 yrs
BSc studentBSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
Only SoPra
AI in workflow
Help me out with writing or correcting code if I get stuck.
Task 1
Coding
Comprehension Quest.
Q1: 4 · Q2: 0 · Q3: 0 · Total 4/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Very Difficult (5)
Mental effort
(7) High mental effort
Confidence
Not at all confident (1)
What were the biggest difficulties you encountered when completing your task (if any)?
Had difficulties understanding how the different components worked together. I also had some trouble with the javascript syntax, since I didn't work with it very often.
Participant P270502AI Group
Task 1 completed
25 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
AI in workflow
explain code snippets
Task 1
Coding
Comprehension Quest.
Q1: 0 · Q2: 4 · Q3: 2 · Total 6/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Easy (2)
Mental effort
(3) Low mental effort
Confidence
Moderately confident (3)
What were the biggest difficulties you encountered when completing your task (if any)?
proof-reading ChatGPT
Task 2
Coding
Comprehension Quest.
Q1: 2 · Q2: 0 · Q3: 0 · Total 2/12
Subjective ReportingParticipant's subjective reporting after working on the second task
Task Difficulty
Medium (3)
Mental effort
(6) Rather high mental effort
Confidence
Slightly confident (2)
What were the biggest difficulties you encountered when completing your task (if any)?
understanding how to set up the auth.controller properly with the correct typer. Actually, most of the code I would take very long to understand without ChatGPT explanations or even solutions from Task 1
Participant P270503No AI Group
25 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
I worked as a software engineer after graduating from university, focusing on a web development project where I extensively used React and TypeScript. During my undergraduate studies, I also worked on several web development projects using various technologies.
AI in workflow
I generally use generative AI to understand pieces of code, fix bugs, or take care of routine tasks like creating plots.
Task 1
Coding
Comprehension Quest.
Q1: 4 · Q2: 4 · Q3: 2 · Total 10/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Difficult (4)
Mental effort
(7) High mental effort
Confidence
Moderately confident (3)
What were the biggest difficulties you encountered when completing your task (if any)?
I was stuck at one point and couldn't move any further. Since the codebase is large, it's hard to find the solution in such a short time. Also, not using LLMs like ChatGPT was another challenge because it's a inevitable part of the coding process nowadays.
Participant P280501AI Group
25 yrs
BSc studentBSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
Webdev, React and TS but not in professional, only hobby professional setting low code solutions (framer, ...)
AI in workflow
Daily
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 2 · Total 8/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Medium (3)
Mental effort
(6) Rather high mental effort
Confidence
Moderately confident (3)
What were the biggest difficulties you encountered when completing your task (if any)?
Getting to know the project and balance between how much AI and how much DIY
Participant P280502No AI Group
Task 1 completed
24 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
Numerous projects in Svelte: -dashboard for visualizing realtime (mqtt) data (location of bots on a map, and reservations) -frontend for a sync system between a (shopify) CMS and a book-keeping system thru their APIs (graphql for the former and REST for the latter). Interface was for visualizing applied changes, as well as initiating syncs. There's other small personal projects that don't involve any frameworks (site for viewing an OTP for a shared account w/ a friend, site for visualizing internet speed test results, visualizing match scores and placing bets between my friends, visualizing the status of self hosted services, etc).
AI in workflow
Coming up with boilerplate code for common patterns (e.g typescript types from arbitrary schemas), checking if there's a simpler refactor possible, etc.
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 2 · Total 8/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Medium (3)
Mental effort
(6) Rather high mental effort
Confidence
Very confident (4)
What were the biggest difficulties you encountered when completing your task (if any)?
Unfamiliarity with the tech stack
Task 2
Coding
Comprehension Quest.
Q1: 2 · Q2: 2 · Q3: 0 · Total 4/12
Subjective ReportingParticipant's subjective reporting after working on the second task
Task Difficulty
Medium (3)
Mental effort
(7) High mental effort
Confidence
Very confident (4)
What were the biggest difficulties you encountered when completing your task (if any)?
Same as before, unfamiliarity with the tech stack and the codebase
Participant P280701No AI Group
Task 1 completedTask 2 completed
27 yrs
MSc studentMSc student in Computer Science (or related field)
Participant Profile
AI usageHow a participant uses generative AI (e.g. Copilot, ChatGPT) to generate new code
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Web dev experience
Yes: I worked professionally for ~4 years at a software company and primarily worked on web application technologies (React, Flask). I also have some experience in a web development course in my undergrad.
AI in workflow
Mainly to generate scripts or visualization code, and sometimes to implement small methods or small modifications to existing code.
Task 1
Coding
Comprehension Quest.
Q1: 2 · Q2: 4 · Q3: 2 · Total 8/12
Subjective ReportingParticipant's subjective reporting after working on the first task
Task Difficulty
Easy (2)
Mental effort
(6) Rather high mental effort
Confidence
Extremely confident (5)
What were the biggest difficulties you encountered when completing your task (if any)?
I was unfamiliar with how to do specific things in the specific technologies, and which parts were connected (called each other)
Task 2
Coding
Comprehension Quest.
Q1: 2 · Q2: 2 · Q3: 0 · Total 4/12
Subjective ReportingParticipant's subjective reporting after working on the second task
Task Difficulty
Very Easy (1)
Mental effort
(3) Low mental effort
Confidence
Extremely confident (5)
What were the biggest difficulties you encountered when completing your task (if any)?
None