The course will closely follow the chapters of the book "Build a Large Language Model (From Scratch)":

1 Understanding large language models
2 Working with text data
3 Coding attention mechanisms
4 Implementing a GPT model from scratch to generate text
5 Pretraining on unlabeled data
6 Fine-tuning for classification
7 Fine-tuning to follow instructions

All practical examples will be done in Python. Laptops in class are required.

After completing this course students will have an in-depth knowledge of the architecture and implementation of modern large language models. Apart from that, they will be able to apply large language models to solve practical problems, such as text classification and question answering.

The rules on the attendance of a Continuous Assessment Course (PI) apply.

At least 80% attendance (physical presence, WU Check-in) is required. Students who fail to meet the attendance requirement will be de-registered from the continuous assessment course with a “fail” grade.

The course will combine theoretical material with hands-on excercises. The instructor will present the course content using the slides; and the students will have to practice implementation of the introduced concepts in class.

The final grade will be computed on the basis of:

• Short multiple choice quizzes after each lecture via LEARN 70%
• Programming project, presentation in final unit 30%

For the programming project participants select a topic in coordination with the lecturer. The project involves application of a large language model to a practical task, such as text classification or question answering.

Grades:

1: =>90%

2: =>80%

3: =>70%

4: =>60%

Please note that grades are assigned strictly as above, without exception: no 'rounding', no extension of deadlines, no additional quizzes or assignments.

The use of AI-based tools like chatGPT for generating the text or code of the final programming project, or the answers to MC questions, is not allowed.

Successful conclusion of the course 1 of SBWL Data Science.

Please be aware that, for all courses in this SBWL, registration is only possibly for students who successfully have completed the entry course (Einstieg in die SBWL: Data Science).

Please log in with your WU account to use all functionalities of read!t. For off-campus access to our licensed electronic resources, remember to activate your VPN connection connection. In case you encounter any technical problems or have questions regarding read!t, please feel free to contact the library at readinglists@wu.ac.at.

A working knowledge of the Python programming language is expected.