Crafting new questions for exams and quizzes can be tedious and time-consuming for educators. The time required varies based on factors like subject matter, question types, experience level, and class level. Multiple-choice questions require substantial time to generate quality distractors and ensure a single unambiguous answer, and composing effective true-false questions demands careful effort to avoid vagueness and assess deeper understanding. Creating high-quality assessment questions of any format necessitates meticulous attention to detail from educators in order to produce fair and valid student evaluations. To streamline this cumbersome process, we propose an automated exam generation solution based on Amazon Bedrock.

In this post, we explore how to build an application that generates tests tailored to your own lecture content. We cover the technical implementation using the Anthropic Claude large language model (LLM) on Amazon Bedrock and AWS Lambda deployed with the AWS Serverless Application Model (AWS SAM). This solution enables educators to instantly create curriculum-aligned assessments with minimal effort. Students can take personalized quizzes and get immediate feedback on their performance. This solution simplifies the exam creation process while benefiting both teachers and learners.

Amazon Bedrock

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading artificial intelligence (AI) companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon using a single API, along with a broad set of capabilities you need to build generative AI applications with security, privacy, and responsible AI. In this post, we focus on a text generation use case, and can choose from Amazon Titan Text G1 and other models on Amazon Bedrock, including Anthropic Claude, AI21 Labs Jurassic, Meta Llama 2, and Cohere Command.

With the ability to scale up to 200,000-token context windows, Anthropic Claude v2.1 on Amazon Bedrock is our preferred choice for this post. It is typically helpful when working with lengthy documents such as entire books. When we talk about tokens, we refer to the smallest individual “atoms” of a language model, and can varyingly correspond to words, subwords, characters, or even bytes (in the case of Unicode). For Anthropic Claude on Amazon Bedrock, the average token is about 3.5 English characters. The 200,000 tokens supported by Anthropic Claude v2.1 on Amazon Bedrock would be equivalent to roughly 150,000 words or over 500 pages of documents.

This post demonstrates how to use advanced prompt engineering to control an LLM’s behavior and responses. It shows how to randomly generate questions and answers from lecture files, implemented as a simple serverless application.

Solution overview

The following diagram illustrates the application architecture. We distinguish two paths: the educator path (1) and the learner path (2).

As first-time users, both educator and learner need to complete the sign-up process, which is done by two separate Amazon Cognito user pools. For the educator, when the sign-up is complete, Amazon Cognito invokes the Lambda function called CognitoPostSignupFn to subscribe the educator to an Amazon Simple Notification Service (Amazon SNS) topic. The educator must approve the subscription to this topic in order to be notified by email with the scorecard of each learner who will be taking the generated exam.

Figure 1: Architectural diagram of the exam generator application

The workflow includes the following steps:

The educator opens the landing page for generating an exam under the domain gen-exam.<your-domain-name> through Amazon Route 53, which redirects the request to the Application Load Balancer (ALB).

1.1 The ALB communicates with Amazon Cognito to authenticate the educator on the educator user pool.

1.2 The educator uploads a lecture as a PDF file into the exam generation front-end.

1.3 The Amazon Elastic Container Service (Amazon ECS) container running on AWS Fargate uploads the file to Amazon Simple Storage Service (Amazon S3) in the Examgen bucket under the prefix exams.

1.4 The S3 bucket is configured using event notification. Whenever a new file is uploaded, a PutObject is activated to send the file to the ExamGenFn Lambda function.

1.5 The Lambda function ExamGenFn invokes the Anthropic Claude v2.1 model on Amazon Bedrock to generate exam questions and answers as a JSON file.

1.6 The Amazon Bedrock API returns the output Q&A JSON file to the Lambda function.

1.7 The ExamGenFn Lambda function saves the output file to the same S3 bucket under the prefix Questions-bank. (You can choose to save it to a different S3 bucket.)

1.8 The ExamGenFn Lambda function sends an email notification to the educator through the SNS topic to notify that the exam has been generated.

The learner opens the landing page to take the exam under the domain take-exam.<your-domain-name> through Route 53, which redirects the request to the ALB.

2.1 The ALB communicates with Amazon Cognito to authenticate the learner on the learner user pool.

2.2 The learner accesses the frontend and selects a test to take.

2.3 The container image sends the REST API request to Amazon API Gateway (using the GET method).

2.4 API Gateway communicates with the TakeExamFn Lambda function as a proxy.

2.5 The Lambda TakeExamFn function retrieves from S3 bucket under the prefix Questions-bank the available exam in JSON format.

2.6 The JSON file is returned to API Gateway.

2.7 API Gateway transmits the JSON file to the ECS container in the front-end.

2.8 The container presents the exam as a UI using the Streamlit framework. The learner then takes the exams. When the learner is finished and submits their answers, the ECS container performs a comparison between the answers provided and the correct answers, and then shows the score results to the learner.

2.9 The ECS container stores the scorecard in an Amazon DynamoDB table.

2.10 The Lambda DynamoDBTriggerFn function detects the new scorecard record on the DynamoDB table and sends an email notification to the educator with the learner’s scorecard.

This is an event-driven architecture made up of individual AWS services that are loosely integrated with each other, with each service handling a specific function. It uses AWS serverless technologies, allowing you build and run your application without having to manage your own servers. All server management is done by AWS, providing many benefits such as automatic scaling and built-in high availability, letting you take your idea to production quickly.

Prerequisites

In this section, we go through the prerequisite steps to complete before you can set up this solution.

Enable model access through Amazon Bedrock

You can add access to a model from the Amazon Bedrock console. For this walkthrough, you need to request access to the Anthropic Claude model on Amazon Bedrock. For more information, see Model access.

Install the necessary packages

You need to install the following:

The AWS Command Line Interface (AWS CLI), an open source tool that enables you to interact with AWS services using commands in your command line shell. For instructions, see Install or update to the latest version of the AWS CLI.
The AWS SAM CLI, which is your toolkit for building and running your serverless application on AWS.
Streamlit, an open source Python framework for building the front-end.
The Docker engine.
Python.
Git.

Register a DNS domain and create certificates

If you don’t already have a DNS domain registered, you need to create one in order to not expose the DNS of your ALB. For instructions, refer to Registering a new domain.

You also need to request two public certificates, one for each front-end: gen-exam.<your-domain-name> and take-exam.<your-domain-name>. Refer to Requesting a public certificate to request a public certificate on AWS Certificate Manager.

Save the values for genCertificateArn and takeCertificateArn.

If you want to build the app in a development environment without using your own domain, you can uncomment the following section in the sam template:

# un-comment if you need to test with HTTP traffic and no certifcate
# ExamGenALBHTTPListener:
# Type: AWS::ElasticLoadBalancingV2::Listener
# Properties:
# LoadBalancerArn: !Ref ExamGenALB
# Protocol: HTTP
# Port: 80
# DefaultActions:
# – Type: forward
# TargetGroupArn: !Ref ExamGenTG

Chain-of-Thought (CoT) Prompting

Before we embark on constructing the app, let’s delve into prompt engineering. We use Chain-of-Thought (CoT) Prompting, which allows the model to break down complex reasoning into smaller, more manageable steps. By providing the AI with intermediate prompts that guide its reasoning process step by step, CoT prompting enables the model to tackle sophisticated reasoning tasks. Guiding the AI through an analytical chain of thought in this way allows it to develop complex reasoning capabilities that would otherwise be beyond its unaided abilities.

In the ExamGenFn Lambda function, we use the following prompt to guide the model through reasoning steps. You can change the prompt and give it different personas and instructions, and see how it behaves.

template_instruction = f”””Human:
You are a teacher during examination time and you are responsible for creating exam questions from the student study book.
Before creating the questions
– Analyze the book found between <exam_book> </exam_book> tags, to identify distinct chapters, sections, or themes for question generation.
– For true/false questions, select statements that can be clearly identified as true or false based on the book’s content.
– For MCQs, develop questions that challenge the understanding of the material, ensuring one correct answer and {n_mcq_options-1} distractors that are relevant but incorrect.
– Randomize the selection of pages or topics for each run to generate a new set of questions, ensuring no two sets are identical.
Please provide the questions in this format exactly for MCQ:
– The output should be like
“question”: “What is the colour of the car in the book?”,
“options”: [“Blue”, “Green”, “Yellow”, “Grey”],
“correct_answer”: “Yellow”
For True/False:
– the output should be like
“question”: “is the sky Blue?”,
“options”: [“True”, “False”],
“correct_answer”: “True”

Generate {n_tfq} true/false and {n_mcq} multiple-choice questions (MCQs) ensuring each question pertains to different pages or topics within the book. For MCQs, provide [n_mcq_options] options for each question. Focus on creating unique questions that cover a broad spectrum of the book’s content, avoiding repetition and ensuring a diverse examination of the material. Use the following guidelines:

1. True/False Questions:
– Craft each true/false question based on factual statements or key concepts from the book.
– Ensure each question spans a wide range of topics to cover the book comprehensively.

2. Multiple-Choice Questions (MCQs):
– Formulate each MCQ to assess understanding of significant themes, events, or facts.
– Include {n_mcq_options} options per MCQ, making sure one is correct and the others are plausible but incorrect.
– Diversify the content areas and pages/topics for each MCQ to avoid overlap and repetition.
“””

Build the exam generator application

The application presented in this post is available in the following GitHub repo with the building blocks code. Let’s start with a git pull on the repo.

We recommend using temporary credentials with the AWS CLI to make programmatic requests for AWS resources using the AWS CLI.

Build the front-end using Streamlit and Docker

You build two containers, one for generating exams and one for taking exams. Let’s start with building the generating exam Docker image:

Go to the following path in the repo and build your Docker image:

user@exam-gen ~ % cd exam-gen-ai-blog/frontend/generate-exam-fe

user@exam-gen generate-exam-fe % docker build -t <your-image-name>:tag .

Authenticate the Docker CLI to Amazon Elastic Container Registry (Amazon ECR):

aws ecr get-login-password –region <your-region> | docker login –username AWS –password-stdin <your-account-id>.dkr.ecr.<your-region>.amazonaws.com

Create a new repository in Amazon ECR:

aws ecr create-repository –repository-name <your-repository-name>

Tag your Docker image with the ECR repository URI:

docker tag <your-image-name>:tag your-account-id.dkr.ecr.<your-region>.amazonaws.com/<your-ecr-repository>:tag

Push your tagged Docker image to your ECR repository:

docker push <your-account-id>.dkr.ecr.<your-region>.amazonaws.com/<your-ecr-repository>:tag

Navigate to this path in the repo to build your Docker image for taking the exam:

user@exam-gen ~ % cd exam-gen-ai-blog/frontend/take-exam-fe

Because the authentication and the ECR repo are already done, run directly the following command:

user@exam-gen take-exam-fe % docker build -t <your-image-name>:tag .

docker tag <your-image-name>:tag your-account-id.dkr.ecr.<your-region>.amazonaws.com/<your-ecr-repository>:tag

docker push <your-account-id>.dkr.ecr.<your-region>.amazonaws.com/<your-ecr-repository>:tag

Copy the values for GenExamImageUri and TakeExamImageUri.

Now that you have both containers ready to run, let’s build the rest of the components using AWS SAM.

Build solution components with AWS SAM

AWS SAM consists of two parts:

AWS SAM template specification – An open source framework that you can use to define your serverless application infrastructure on AWS
AWS SAM CLI – A command line tool that you can use with AWS SAM templates and supported third-party integrations to build and run your serverless applications

For further information, refer to Using the AWS Serverless Application Model (AWS SAM).

Go to the home directory user@exam-gen ~ % cd exam-gen-ai-blog and run the sam build command.

Before you run sam deploy, be aware of the following:

The ECS containers are deployed on Fargate, which needs a VPC with two subnets in different Availability Zones. We use the default VPC for simplicity. You can create your own VPC or use an existing one in your AWS account and update the sam template. To list your VPC IDs and subnets within a selected VPC ID, run the following commands to extract your VpcId and your two SubnetId:

aws ec2 describe-vpcs
aws ec2 describe-subnets

GenExamCallbackURL (for generating exam) and TakeExamCallbackURL (for taking exam) are used by Amazon Cognito. They are URLs where the user is redirected to after a successful sign-in.

Now let’s deploy the sam template:

sam deploy –stack-name <your-stack-name> –guided
–parameter-overrides
DefaultVPCID=”your-default-vpc-id”
SubnetIdOne=”your-subnet-one-id”
SubnetIdTwo=”your-subnet-two-id”
genCertificateArn=”arn:aws:acm:<your-region>:<your-account-id>:certificate/<your-certificate-id>”
takeCertificateArn=”arn:aws:acm:<your-region>:<your-account-id>:certificate/<your-certificate-id>”
GenExamImageUri=”<your-gen-image-uri>”
TakeExamImageUri=”<your-take-image-uri>”
GenExamCallbackURL=”gen-exam.<your-domain-name>”
TakeExamCallbackURL=”take-exam.<your-domain-name>”
NotificationEmail=”your-email-address@example.com”
–capabilities CAPABILITY_NAMED_IAM
#Shows you resources changes to be deployed and require a ‘Y’ to initiate deploy
Confirm changes before deploy [Y/n]: n
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]: y
#Preserves the state of previously provisioned resources when an operation fails
Disable rollback [Y/n]: n
Save arguments to configuration file [Y/n]: n

Looking for resources needed for deployment:
Creating the required resources…

Successfully created!

You can follow the creation on the AWS CloudFormation console.

This following video demonstrates running the sam build and sam deploy commands.

Figure 2: SAM build and SAM deploy execution

The final step is to get the DNS names for the deployed ALB, map them to the certificate domains names in Route 53, and add them as a CNAME record.

Test the solution

You can use your browser to test the solution.

Navigate to gen-exam.<your-domain-name>.

You’ll receive an email with a confirmation code.

Enter the verification code and choose Confirm account.

Once verified, you will land on a page to generate your quiz.

Choose the amount of multiple choice and true/false questions you want to generate, then choose Browse files to upload an input file.

For this example, we use the whitepaper AWS Cloud Adoption Framework: Security Perspective as our input file. We generate four multiple-choice questions and one true/false question.

Confirm your subscription to the SNS topic (you’ll receive an email).

Then you’ll receive an email confirming the exam has been generated.

Switch to take-exam.<your-domain-name>, and you’ll find the exam on the dropdown menu.

Choose the exam, then choose Load quiz.

Then you can take the exam and choose Submit to display the results.

The educator will receive an email with the scorecard of the learner.

You have just built a simple application that randomly generates questions and answers from uploaded documents. Learners can take the generated exams and educators can receive scorecards via email when tests are complete. The integration with the DynamoDB table allows you to store the responses on a long-term basis.

Expanding the solution

There are many possibilities to build on top of this and create a fully featured learning and testing application. One area of expansion is uploading multiple documents at once. As of this writing, users can only upload one document at a time, but support for bulk uploads would improve efficiency and make it easier to work with large sets of source materials. Educators could be empowered to gather and upload content from various documents and websites as source material for questions. This provides greater flexibility compared to using a single document. Moreover, with a data store, they could view and analyze learner answers via a scorecard interface to track progress over time.

Clean up

It’s important to clean up your resources in the following order:

On the Amazon S3 console, empty the bucket by deleting any files and folders.

On the AWS CloudFormation console, delete the stack.

Conclusion

In this post, we showed how to build a generative AI application powered by Amazon Bedrock that creates exam questions using lecture documents as input to support educators with an automated tool to continuously modernize quiz material and improve learners’ skills. Learners will be able to take the freshly generated exam and get the score results. With the capabilities of Amazon Bedrock and the AWS SAM, you can increase educators’ productivity and foster student success.

For more information on working with generative AI on AWS for education use cases, refer to Generative AI in education: Building AI solutions using course lecture content.

About the Authors

Merieme Ezzaouia is a Solutions Architect at AWS dedicated to the public sector. She helps customers in education and sports turn their concepts into tangible solutions, develop new services, and foster innovation. Beyond work, Merieme’s passions include gardening, traveling the world, and reading.

Mohammed Reda is a Solutions Architect at Amazon Web Services. He helps UK schools, universities, and EdTech companies adopt cloud technologies, improve their educational offerings, and innovate on AWS. Outside of work, Mohammed enjoys running and watching cooking shows.

Categorized in:

Technology