devtrackers.gg
7 months
ago -
EVE Online
-
Direct link
Transcript (by Youtube)
| 4s | hello everyone |
|---|---|
| 7s | Welcome to The Fan Fest where we |
| 10s | celebrate 20 years of Eve online |
| 12s | 20 years such a long time for an MMO |
| 16s | or for any video game to exist |
| 17s | continuously really |
| 20s | my name is CCP aporia I'm a principal |
| 22s | programmer on the platform Technology |
| 24s | Group |
| 25s | and I'm here to talk about how we |
| 28s | undertake maintenance projects on our |
| 31s | code base how we future proof our code |
| 33s | base in order to keep the game running |
| 36s | for another 10 hundred or a thousand |
| 39s | years who knows right e forever |
| 41s | and I'm here today with my colleague CCP |
| 44s | Queen B |
| 47s | thank you yeah hi hello great to see you |
| 50s | and I'm so happy to be here actually |
| 52s | I've been with CCP more or less since |
| 55s | 2007 but I haven't had a chance to |
| 58s | attend Fun Fest for various reasons |
| 60s | since 2015 so I went thinking back when |
| 63s | I realized this it's just crazy so I'm |
| 65s | very very excited |
| 67s | to be here today and happy for this |
| 69s | chance to tell you about what we're |
| 70s | doing |
| 71s | I'm a software engineer |
| 74s | and I've been in various engineering and |
| 77s | management roles at CCP over the years |
| 79s | and my current role is technical manager |
| 81s | which allows me to do both management |
| 84s | and programming |
| 86s | I absolutely |
| 88s | enjoy a lot writing code I also enjoy a |
| 93s | lot sometimes even more deleting code |
| 95s | and you'll hear more about that later |
| 98s | yeah |
| 99s | some of you know what I'm talking about |
| 102s | this is our goal like ccpuria mentioned |
| 106s | like e forever right |
| 109s | and towards that end we work on various |
| 111s | projects for years and even decades and |
| 115s | we maintain operate and actively develop |
| 119s | code that can be up to and sometimes |
| 121s | even over 20 years old |
| 123s | so what sort of implications does that |
| 126s | have for the engineers what sort of |
| 129s | challenges does it present |
| 132s | well one challenge what I like to think |
| 134s | of as keeping the kitchen clean |
| 136s | and just this is really quite similar to |
| 139s | when you're working in the kitchen you |
| 141s | know you should you know put things away |
| 143s | keep it tidy as you go it's a lot more |
| 144s | enjoyable faster and easier to work in a |
| 147s | clean kitchen |
| 149s | and yet that's not always how it is or I |
| 153s | don't know about you guys but my kitchen |
| 155s | at least sometimes gets like really |
| 157s | messy and why is that well it's just not |
| 161s | always practical if you think about it |
| 162s | if you were to wash every single like |
| 165s | little knife or spoon or whatever and |
| 166s | put it away as we were working still |
| 168s | things would start to take a long time |
| 170s | and sometimes we just have a lot going |
| 173s | on we're cooking up you know we have a |
| 175s | big dinner party and the guests are |
| 176s | coming and things are cooking in five |
| 178s | different pots and pans |
| 181s | and then the guests have arrived the |
| 182s | food is ready we put it out and we close |
| 184s | the door on the mess |
| 185s | and we have a lovely evening a lot of |
| 188s | fun wake up the next morning you have a |
| 190s | choice do you spend time tidying up or |
| 193s | do you just you know grab a bowl of |
| 194s | cereal take it to the living room and |
| 196s | close it or on the mess again |
| 198s | and you can keep this up for some time |
| 201s | you know you can just live on cereal and |
| 204s | and maybe toast for a while |
| 206s | and now if you were going to and this is |
| 209s | not a very realistic scenario but let's |
| 211s | say you were going to just move out of |
| 213s | the house just abandon it and you didn't |
| 215s | care one bit how you left it then you |
| 217s | probably wouldn't even bother |
| 219s | but if you're going to live in that |
| 221s | house and work in that kitchen for years |
| 223s | you're gonna have to tackle that mess |
| 225s | one day or eventually |
| 228s | and working in code that is heavily |
| 232s | burdened with technical debt it does |
| 234s | sometimes feel like this |
| 235s | and what happens is I'm sure some of you |
| 238s | are familiar with this and can relate |
| 239s | you start to like make clean a tiny |
| 243s | little spaces to work in and you try not |
| 245s | to touch too much because you know and |
| 246s | not make any sudden moves because things |
| 248s | might just start to topple over you know |
| 251s | as an analogy I just want to say it's |
| 254s | not of course entirely accurate and not |
| 256s | fair either because it gives the |
| 258s | impression that the developers who made |
| 260s | the mess you know they were just being |
| 262s | lazy or sloppy or whatever |
| 264s | and generally this is not the case |
| 267s | um usually people are working given a |
| 269s | certain set of constraints there's |
| 271s | almost always time constraints |
| 274s | um there's backwards compatibility to |
| 277s | deal with there's the |
| 279s | just the technique attack that is |
| 282s | available at the time Frameworks that |
| 284s | you have to work with |
| 285s | you have maybe you know you work based |
| 289s | on the knowledge that you have at the |
| 291s | time I'm sure those of you who are |
| 293s | developers know the feeling when you're |
| 294s | done with the project you know exactly |
| 296s | how you would do it if you would you |
| 298s | know start over so it's all these things |
| 301s | generally people are really doing their |
| 303s | best and it's very easy to judge when |
| 306s | you look at old code but you have to |
| 308s | look back with kindness because |
| 309s | in my experience what almost and I think |
| 313s | all developers really want most of all |
| 314s | is just to write beautiful code that's |
| 316s | what we're all trying to do |
| 319s | um but |
| 321s | even if we had nothing but beautiful |
| 323s | perfect coat that did everything we |
| 325s | wanted |
| 326s | we still have another challenge and that |
| 329s | is that the world is constantly changing |
| 332s | around this and like if you just look |
| 334s | back if you think about it 20 years ago |
| 336s | the technology at the time and the |
| 338s | changes since then it's just it's |
| 341s | mind-blowing really and I came across |
| 344s | this image and I don't know like how |
| 346s | much of it you can see |
| 347s | it sort of goes through a little bit the |
| 350s | evolution and devices and programming |
| 354s | languages here and operating systems |
| 357s | all the things this this is just one |
| 360s | small piece of it this doesn't even |
| 361s | begin to cover uh Cloud systems software |
| 365s | as a service infrastructures code all |
| 367s | these things |
| 369s | and this is uh so we have to keep |
| 373s | evolving and adapting and we could |
| 375s | probably spend our time doing nothing |
| 377s | but that and falling back to the kitchen |
| 380s | analogy that would be a little bit like |
| 382s | changing the countertops every week and |
| 384s | we couldn't be you know cooking anything |
| 385s | because the kitchen isn't a constant |
| 387s | state of renovation and again we're |
| 389s | eating nothing but cereal and toast |
| 391s | so |
| 393s | um it's a balancing act when do we make |
| 397s | do with what we already have and works |
| 399s | and when is it time to spend time |
| 403s | adapting to new tech possibly even |
| 406s | rewriting or replacing entire components |
| 410s | and with that I'm going to give it back |
| 413s | over to ccporia |
| 415s | thank you Queen B |
| 417s | so as you can see a lot of things have |
| 418s | changed and as Queen B just mentioned |
| 421s | adapting to various Technologies or |
| 424s | environmental changes to happen in this |
| 425s | time there's always a careful Balancing |
| 427s | Act |
| 428s | and the first step when approaching a |
| 431s | new project to adapt to the changing |
| 433s | realities is to effectively take a look |
| 436s | at the uh |
| 439s | history |
| 442s | the pointer isn't working |
| 446s | can you |
| 447s | use the mouse please |
| 464s | this is the reason why we have to do |
| 466s | maintenance work on technology |
| 469s | so that |
| 470s | so that these things happen less |
| 472s | regularly |
| 474s | so |
| 476s | while we're waiting for the computer to |
| 478s | uh come back to life |
| 482s | as we said the first thing |
| 485s | that we always do |
| 487s | when we have to tackle new updates and |
| 490s | changing environments is that we take a |
| 492s | look at the history |
| 493s | in this specific case unfortunately you |
| 495s | can't see the slide |
| 497s | but we decided to upgrade our python |
| 499s | interpreter version that we're running |
| 502s | basically if online since 2009 has been |
| 504s | running on a version of Python 2.7 |
| 507s | a version of stackless Python 2.7 |
| 510s | and some of you may or may not know but |
| 513s | in 2009 it was also already announced |
| 515s | the 2.7 would be the last python 2 |
| 519s | version that the future would be python |
| 521s | 3. |
| 522s | and |
| 524s | that you know in a few years no more |
| 527s | support for python 2 will be happening |
| 530s | at that point in time actually happened |
| 532s | three and a half years ago |
| 534s | so we are effectively left with the |
| 537s | choice of maintaining python 2 ourselves |
| 541s | or to you know move it move our |
| 544s | technology into the future and adapt to |
| 545s | Python 3 which was a really big change |
| 547s | in the ecosystem for some of you that |
| 550s | may know it |
| 552s | um |
| 553s | so to tackle this project we need to |
| 555s | look at the history of python usage at |
| 556s | CCP |
| 558s | it is a very crucial part of our |
| 559s | technology stack very early on in |
| 561s | development |
| 563s | CCP had realized |
| 565s | that using python allowed for a very |
| 568s | quick development speed |
| 569s | it's a very expressive language it's |
| 571s | really quick to get things rolling with |
| 574s | it some of you may be aware of python |
| 577s | and use it in daily life so if you just |
| 579s | open a command line or a shell and type |
| 582s | in the python command you get an |
| 583s | interpreter |
| 584s | you type in a small instruction and it |
| 588s | just instantly executes right you get |
| 589s | really quick feedback that's really |
| 591s | great it's a quick iteration cycle and |
| 594s | that just means you get to try out ideas |
| 596s | faster you develop faster |
| 599s | the other part was that python has a |
| 602s | very expressive language also allowed |
| 604s | non-programmers to contribute to the |
| 607s | development efforts |
| 609s | yes we are there thank you |
| 615s | so it did |
| 617s | this low entry barrier |
| 619s | enabled game designers to write up a |
| 622s | little effect that they needed to have |
| 624s | technical artists write up a little |
| 626s | animation that they needed and overall |
| 628s | you know for a company that's about to |
| 631s | roll out a game that's a really great |
| 633s | benefit to have because time to Market |
| 636s | is Money in the Bank |
| 641s | as a consequence for a long time CCP was |
| 643s | very very deeply involved in the python |
| 647s | ecosystem CCP was a sponsor on many |
| 650s | pycon conference instances |
| 653s | we contributed a lot to C Python and |
| 656s | stackless python |
| 658s | and we even employed two of the core |
| 660s | maintainers of stackless python for |
| 662s | several years at the company |
| 665s | last but not least |
| 667s | we were the company that ported python |
| 670s | to the PlayStation 3. |
| 672s | that is quite the impressive technical |
| 674s | feat |
| 676s | um |
| 676s | the PlayStation 3 had 256 megabytes of |
| 679s | memory |
| 680s | the phone you're holding there that has |
| 682s | more than 256 megabytes of memory |
| 684s | and the python interpreter itself |
| 686s | without loading any code without loading |
| 688s | any modules of the standard library is |
| 690s | by default 25 megabytes of memory |
| 693s | so we had only 200 megabytes roughly to |
| 696s | fit in all the textures game assets data |
| 698s | files and the actual game logic right so |
| 702s | in short we were quite involved we put |
| 705s | off quite many magic things with python |
| 707s | but you know we need to keep using it |
| 711s | because of that and |
| 713s | we needed reasons to upgrade because |
| 716s | this is a situation we were in right we |
| 718s | have all this all these features in the |
| 720s | game |
| 721s | and a good foundation but there's this |
| 723s | tiny really important pillar that's |
| 724s | Python 2.7 and that's slowly crumbling |
| 726s | away |
| 728s | so as I mentioned previously while we're |
| 731s | waiting for the computer to come back to |
| 732s | life |
| 733s | python 2 was officially end of life |
| 735s | January 2020. |
| 738s | this kind of prompted us to the reality |
| 741s | that we have to adapt |
| 744s | when we hire new people |
| 747s | they usually don't have a lot of |
| 748s | experience of python 2. that's okay you |
| 751s | know the world is using Python 3 for the |
| 753s | most part and you know we're using |
| 755s | stackless python as well so that's |
| 756s | another complication in the in the book |
| 760s | um one thing we then realized as we were |
| 762s | developing the native net client |
| 764s | recently is that |
| 767s | porting |
| 768s | python 2 projects two new hardware like |
| 771s | the Apple silicon chip |
| 773s | is not quite straightforward python 2 |
| 776s | has no concept of that |
| 778s | you know they don't know there's an |
| 779s | arm64 that runs Mac OS so we actually |
| 782s | had to spend quite some resources to get |
| 785s | Eve running on that Hardware as well |
| 787s | with python 2. |
| 790s | overall the python ecosystem has moved |
| 792s | on well this ties a little bit in with |
| 794s | new hires not knowing python 2. but this |
| 798s | is also a lot of libraries that we are |
| 800s | using a lot of packages from the python |
| 802s | ecosystem that we are using and that we |
| 803s | also have contributed back to well they |
| 805s | are slowly leaving python 2 behind or |
| 808s | have left python 2 behind this makes |
| 810s | certain upgrades for us really difficult |
| 812s | because we need to be very careful about |
| 815s | the version of the package we can use |
| 817s | and if it's let's say a networking |
| 819s | package and there's a security issue |
| 821s | with it you know that can get quite |
| 823s | cumbersome |
| 826s | and another thing as well is because |
| 829s | Python 3 is a lot more modern on many |
| 832s | fronts it allows us to get rid of some |
| 835s | of the customizations that we had to do |
| 836s | to stackless Python 2.7 |
| 838s | I mean there are minor things like |
| 840s | Unicode support right I mean who wants |
| 842s | to play in Korea or Japan or Russia |
| 845s | right no this is important and we had to |
| 847s | go to very Great Lengths in Python 2.7 |
| 849s | to actually support us properly in |
| 851s | Python 3 that's a lot better and easier |
| 854s | another thing is memory allocation |
| 856s | routines that doesn't require us to cut |
| 858s | deeply into it there's a nice interface |
| 860s | for just specifying the functions you |
| 862s | use and so we can just you know benefit |
| 864s | from some changes in Python 3. |
| 866s | that's a great list of of reasons to |
| 868s | upgrade |
| 869s | and with that we started to come up with |
| 872s | something called pre-production where we |
| 874s | want to figure out how can we do this |
| 877s | update you know we really want to we |
| 878s | have good reasons but how can we do it |
| 880s | it's such a fundamental piece of our |
| 882s | technology basically everything depends |
| 885s | on it the graphics engine there's a lot |
| 887s | of python on Leaf load audio engine lots |
| 889s | of python and refilled Network stack |
| 891s | that's all in python as well to some |
| 892s | degree you know it's very crucial that |
| 895s | we approach this carefully so during |
| 896s | pre-production we want to answer a few |
| 899s | questions we want to formulate a few |
| 901s | questions as well like |
| 903s | if we do this upgrade can we simply drop |
| 905s | in Python 3 you know go to the |
| 907s | python.org website download python3 drop |
| 909s | it in the SDK folder compile will that |
| 911s | work well no |
| 915s | it's not that easy but uh |
| 918s | well that was quite obvious to answer |
| 920s | but the next one was then can we mix and |
| 921s | match python two and three |
| 923s | python 2 comes with a future module |
| 926s | that allows you to import certain future |
| 928s | Behavior you know the python guy is |
| 930s | always doing some magic like |
| 931s | anti-gravity but in this case future so |
| 933s | you can write a certain import statement |
| 935s | at the beginning of your python script |
| 936s | and that turns the python script into |
| 939s | behaving like as if it would be running |
| 940s | under Python 3 but almost but very close |
| 943s | and can we maybe make benefit of that |
| 948s | then |
| 949s | last but not least do we know the full |
| 951s | extent of the work that is required |
| 954s | like there is 20 years of History |
| 957s | a lot of time that has passed a lot of |
| 958s | code that has been written |
| 961s | and you know people that have been |
| 964s | working on it have moved on to other |
| 965s | projects other companies or other |
| 967s | positions |
| 969s | and um |
| 971s | the version of status python you're |
| 973s | using is heavily customized do we know |
| 975s | all the customizations |
| 977s | 99 of them are probably annotated or 98 |
| 980s | I ran the numbers once when I was |
| 982s | looking into it but uh you know not all |
| 984s | of the customizations are annotated |
| 987s | so |
| 989s | overall it's a very very complex |
| 991s | situation and it requires us to do a lot |
| 993s | of Investigation of of how we can |
| 995s | actually really go through with the work |
| 998s | that we have to do |
| 1000s | and this investigation sometimes gets |
| 1002s | surprising results |
| 1004s | on that slide you can see |
| 1007s | two main lines the large one the top one |
| 1011s | is the the python code lines the lower |
| 1013s | two ones are C plus code |
| 1017s | um |
| 1018s | and you may be able to see it but here |
| 1020s | is roughly the time when Python 2.7 was |
| 1022s | announced this deprecated |
| 1025s | and since then we have basic quadruple |
| 1028s | the amount of python code we have in our |
| 1029s | code base |
| 1031s | good job |
| 1033s | makes this job easier right so we we |
| 1036s | made some interesting discoveries like |
| 1037s | this and that just meant the whole work |
| 1039s | just gets a little bit more complex |
| 1043s | but because basically everything hinges |
| 1044s | on it and we need to do it we need to be |
| 1046s | very careful |
| 1048s | so with a complex huge problem like this |
| 1051s | what do you do the same thing that |
| 1052s | Evolution has always done and that |
| 1054s | humans also have practiced very |
| 1055s | successfully for a few thousand years |
| 1057s | when faced with a huge problem divide |
| 1059s | and conquer |
| 1061s | we came up with three distinct tracks in |
| 1063s | which we want to split this work |
| 1065s | the first one is the stackless python |
| 1067s | interpreter itself like |
| 1070s | our 2.7 version is heavily customized |
| 1074s | how many of these customizations do we |
| 1076s | need to keep |
| 1077s | and also how can we make sure that these |
| 1079s | customizations that we do have are not |
| 1081s | getting in the way of future updates |
| 1083s | like they just did with the three with |
| 1085s | the version 3 update right now |
| 1088s | so we had to come up with a solution for |
| 1090s | that the second stage is then you have |
| 1092s | just seen the four million lines of |
| 1093s | python code that we have how can we get |
| 1096s | those translated over |
| 1098s | can we just do them wholesale to Python |
| 1101s | 3 right away |
| 1102s | well |
| 1103s | probably but it's not necessarily the |
| 1105s | smartest idea because a lot of the game |
| 1107s | logic Clips in python in the Python |
| 1109s | scripts so any mistake in there |
| 1112s | can have disastrous consequences so |
| 1115s | maybe there's a way for us to be a bit |
| 1116s | more |
| 1118s | um conservative and a bit less risky in |
| 1120s | that approach |
| 1122s | the last one is then migrating our C |
| 1124s | plus plus components that we have |
| 1126s | two Python 3 and |
| 1128s | well that is unfortunately a track where |
| 1131s | we had to do everything |
| 1133s | but with these three tracks identified |
| 1134s | individually we could limit the risk and |
| 1137s | the scope of this project as much as we |
| 1139s | can |
| 1142s | this is good in the sense that it also |
| 1144s | allows us to fail early like if we can't |
| 1146s | get the stackless python interpreter |
| 1148s | up and running properly with our |
| 1150s | constraints you know then we may as well |
| 1152s | just say okay we cannot update to Python |
| 1154s | 3 we need to find other ways of dealing |
| 1156s | with python 2 in the future |
| 1157s | and discovering these sorts of failures |
| 1159s | early in such a project is really |
| 1161s | important because the later you discover |
| 1163s | these sorts of failures the more |
| 1164s | expensive it gets to basically resolve |
| 1167s | them or to fix them like if you would |
| 1169s | have discovered this like a year later |
| 1171s | then |
| 1173s | well a lot of money has been spent |
| 1176s | but in order to then tackle all these |
| 1179s | three tracks that we had set up we |
| 1182s | decided to leverage some expertise |
| 1184s | now the C plus plus code is obviously |
| 1186s | something where the expertise lays |
| 1187s | in-house we have written our physics |
| 1189s | engine we have written our Graphics |
| 1191s | engine we have written our audio engine |
| 1192s | and the network stack and various other |
| 1195s | components that we have and a lot of |
| 1197s | these systems have such Arcane knowledge |
| 1200s | encoded in them that you know |
| 1203s | bringing other people into it to help us |
| 1205s | difficult would be the same amount of |
| 1207s | effort as just doing at work ourselves |
| 1210s | but on the python front you know Python |
| 1213s | 3 has been out for well over a decade |
| 1215s | and there's many companies that help |
| 1217s | with like doing transitions from python |
| 1219s | 2 to python 3. we found really good |
| 1221s | partners in a company called reckon |
| 1223s | digital |
| 1224s | and you know they started helping us out |
| 1227s | and we are very very pleased with the |
| 1228s | results so far that we have seen the the |
| 1230s | guys have been blazing through the |
| 1232s | upgrade process that's kind of |
| 1234s | impressive |
| 1236s | and |
| 1237s | the other thing or the last important |
| 1239s | thing to keep in mind for such complex |
| 1242s | projects is it's really difficult to put |
| 1245s | a time scale on it |
| 1246s | like you can't just say like oh yeah I'm |
| 1248s | just going to put in the Python 3 |
| 1249s | interpreter get our things to comply |
| 1251s | with it and run some scripts on a python |
| 1253s | 2 code to convert it and it's going to |
| 1254s | take me like six months nah |
| 1257s | um there are too many unknowns |
| 1259s | for for these things to happen |
| 1262s | and |
| 1263s | as such we basically decided to set |
| 1266s | certain Milestones that we wanted to |
| 1267s | reach again the idea being fail early |
| 1270s | fail fast |
| 1272s | we have our game engine something called |
| 1274s | the python interpreter mode where we can |
| 1276s | start it up like a python shell that was |
| 1278s | for us the first Milestone where we |
| 1280s | would know if our integration of Python |
| 1282s | 3 would work |
| 1283s | so we set that as the first milestone |
| 1286s | the other two important Milestones that |
| 1287s | we then had |
| 1289s | because they are critical to running the |
| 1290s | game and getting it in your hands |
| 1292s | is getting the server and the client up |
| 1294s | and running now we could have switched |
| 1295s | them around but we opted for the the way |
| 1297s | of first getting the server up and |
| 1299s | running simply because it requires less |
| 1301s | C plus components |
| 1303s | and as we're running two of these three |
| 1305s | tracks that are outlined earlier in |
| 1306s | parallel where I can digital doing the |
| 1308s | python work while we are doing the C |
| 1309s | plus plus work |
| 1311s | that means you know we kind of can keep |
| 1313s | Pace with them |
| 1314s | and we get to see results a bit quicker |
| 1319s | and all of these complexities and |
| 1322s | realities then ended up in a oh that's a |
| 1325s | wrong slide sorry and then up in |
| 1328s | in a situation like this |
| 1331s | um this looks very very busy in the |
| 1333s | graphic |
| 1334s | but effectively |
| 1335s | we have the two tracks right on the top |
| 1337s | we have the python track and the bottom |
| 1339s | we have the espresso's track |
| 1341s | that's actually from a stakeholder |
| 1342s | meeting we had a few weeks ago so this |
| 1343s | is fairly up to date we will focus on |
| 1346s | the project and here you can see that in |
| 1348s | order to isolate our risks a bit more we |
| 1351s | have certain check-in points the python |
| 1353s | code runs in a specific bra the python |
| 1355s | two to three transition runs in a in a |
| 1358s | certain Branch to begin with once we are |
| 1360s | happy with the results there we get that |
| 1362s | into the main line of one or one of our |
| 1364s | products |
| 1365s | this is important because our test |
| 1368s | infrastructure may not necessarily cover |
| 1369s | all of the intrinsic intrinsic details |
| 1371s | that the project has whatever the game |
| 1374s | has and this is then a point where we |
| 1376s | can figure out more of the critical |
| 1378s | Parts well this is where these things |
| 1380s | are going on |
| 1381s | we happily Chuck along and get our c |
| 1383s | plus components ready |
| 1385s | and at one point they are kind of ready |
| 1388s | right and at that point we can start |
| 1390s | getting the python code purely python |
| 1392s | free as well |
| 1394s | get them integrated |
| 1396s | and |
| 1397s | end up with like hopefully something we |
| 1399s | can release at some point |
| 1401s | uh but yeah so these are the |
| 1403s | complexities that we sometimes have to |
| 1405s | deal with and especially in the case of |
| 1406s | this Python 3 project that we have |
| 1408s | approached Queen B is now going to tell |
| 1410s | you about another project we've recently |
| 1412s | been doing |
| 1415s | that is right so we had uh some |
| 1419s | different challenge |
| 1422s | and when I saw this image it really |
| 1424s | resonated with me it really felt like it |
| 1426s | captured it well it's uh made by a |
| 1430s | Belgian visual artist his name is Philip |
| 1432s | desert you can find him on Instagram and |
| 1434s | he makes these fictional photographs of |
| 1437s | mostly architecturally impossible |
| 1438s | buildings |
| 1440s | and what we had was this |
| 1444s | big monolithic monolithic system that |
| 1448s | had been growing over the years and it |
| 1450s | held pretty much everything except the |
| 1453s | server so it had like the payment stuff |
| 1456s | the accounts the sign up the city Keys |
| 1459s | the body program the you know what all |
| 1462s | sorts of things that gathered over the |
| 1465s | years |
| 1466s | we knew some of the pieces weren't |
| 1467s | needed but we didn't really know what |
| 1469s | would happen if we took them out |
| 1471s | and like apodia mentioned we had you |
| 1475s | know people had come and gone there were |
| 1477s | some lost knowledge there |
| 1478s | and frankly people were a bit scared to |
| 1480s | go in there because they didn't know |
| 1482s | where they would end up |
| 1484s | so we knew we had to get out of that |
| 1485s | situation like heading for the future |
| 1488s | and at first we thought we would just |
| 1492s | abandon that |
| 1494s | system you would write a new one it |
| 1496s | would be simple and beautiful and do |
| 1498s | everything we wanted |
| 1500s | and of course that was just an illusion |
| 1502s | it was this we were fantasizing about a |
| 1506s | Greenfield project and by definition |
| 1508s | that is a product where you start from |
| 1509s | scratch |
| 1511s | you have no Legacy systems no |
| 1514s | infrastructures no dependencies quite |
| 1516s | obviously not what we had |
| 1519s | unfortunately we realized this quickly |
| 1521s | we didn't spend a lot of time on it |
| 1523s | I wanted to mention it though because |
| 1525s | this is so common |
| 1527s | um it's so often when you have a |
| 1530s | complicated system lost knowledge |
| 1532s | technical debt all these things you |
| 1534s | think are just be easier to write it |
| 1536s | from scratch I've heard many stories |
| 1538s | like this I'm sure you've some of you |
| 1540s | are familiar with them as well |
| 1542s | um it's almost never the answer exactly |
| 1544s | because it's complicated you don't know |
| 1546s | all the requirements and the system is |
| 1549s | still in active development it's going |
| 1550s | the requirements are going to change as |
| 1552s | you are working |
| 1554s | so beware |
| 1557s | um Plan B we would start cutting out the |
| 1560s | pieces that we knew we didn't need we |
| 1562s | would figure out you know what was |
| 1563s | needed excuse me uh how we could cut |
| 1567s | them out and then at the same time we |
| 1569s | would improve and fix the things that we |
| 1572s | wanted to take forward towards the |
| 1574s | future so we started acting on that and |
| 1577s | we did so mainly in two ways on one hand |
| 1581s | we went in just deliberately removing |
| 1585s | components such as the body program as I |
| 1587s | remember that I mentioned which was a |
| 1589s | precursor to today's recruitment system |
| 1592s | and the recruitment system today is it's |
| 1595s | on just separate independent service but |
| 1598s | about the program at the time was a part |
| 1601s | of this big thing and it had tentacles |
| 1603s | into other things so even if it was no |
| 1605s | longer in use it was still affecting the |
| 1608s | work that we were doing we sometimes had |
| 1610s | to you know work around it there were |
| 1612s | tests automated tests that were ensuring |
| 1614s | that the body program still worked so |
| 1616s | when we were making changes elsewhere we |
| 1618s | sometimes had to you know fix that so |
| 1621s | just we went in with uh components like |
| 1624s | this removing them and like just digging |
| 1627s | deep going removing any dependencies |
| 1630s | then that were Obsolete and so on so |
| 1632s | just try to get some of these |
| 1636s | complications that were no longer |
| 1638s | relevant out of the picture |
| 1641s | and we sort of worked on this alongside |
| 1643s | other development work for a good |
| 1646s | stretch of time at the same time the |
| 1647s | other thing that we did was just to be |
| 1650s | always very mindful about being good |
| 1652s | citizens about when we went in we're |
| 1654s | doing development we were fixing |
| 1656s | something and adding something |
| 1659s | to make sure to leave the code better |
| 1661s | than it was when we came in like when |
| 1664s | you if you have to go in and you have to |
| 1666s | figure out and you have to spend time |
| 1667s | figuring out what's happening |
| 1669s | try to leave it so that when the next |
| 1672s | person come or when you come in six |
| 1674s | months that you don't have to go through |
| 1676s | that process again |
| 1677s | and just as a practical or just a very |
| 1680s | simple example of this |
| 1682s | in the original implementation of a |
| 1685s | payment object we had assumed we would |
| 1688s | be storing some credit card information |
| 1690s | I think it was the last four digits and |
| 1692s | the CVC code |
| 1694s | this never happened we never stored this |
| 1696s | data |
| 1698s | but it was always a part of the payment |
| 1700s | these properties were always there and |
| 1702s | they were always being you know when you |
| 1703s | added something new that that was |
| 1704s | payment you were always dealing with |
| 1705s | this making sure to copy it along and |
| 1708s | when a new programmer came in and they |
| 1709s | would be like are we storing credit card |
| 1712s | information I didn't know that is this |
| 1714s | neat that for backwards compatibility or |
| 1715s | why is this there |
| 1716s | and it's a small thing |
| 1719s | but many small things add up to a lot of |
| 1723s | like just mental burden and confusion |
| 1726s | um |
| 1727s | another just practical example this is |
| 1730s | from a recent change |
| 1733s | the code on the left was deleted in |
| 1735s | favor of that one green line on the |
| 1737s | right |
| 1738s | and this is also example of yeah isn't |
| 1742s | this fun |
| 1743s | a code on the left is also a good |
| 1746s | example I think of how this technical |
| 1748s | depth grows somebody comes in they need |
| 1750s | to add some logic and they don't want to |
| 1751s | touch the other logic because it |
| 1753s | probably needs to be there so I'll just |
| 1754s | squeeze my thing in there |
| 1756s | so rather than do that |
| 1758s | uh the Development question spent some |
| 1761s | time making sure you know that this was |
| 1763s | safe this could be done and simplified |
| 1765s | and then what I had with this change |
| 1768s | if you were a developer you might be |
| 1770s | thinking I hope you're thinking well |
| 1772s | didn't you have tests |
| 1774s | why did you have to spend even time you |
| 1776s | know validating this |
| 1778s | very true and tests are super important |
| 1780s | automated tests |
| 1782s | um thankfully and this has there's |
| 1783s | actually a lot somewhat surprisingly for |
| 1786s | such old code it actually most of the |
| 1788s | system had pretty decent test coverage |
| 1791s | and that often helped us at the same |
| 1793s | time tests are of course themselves |
| 1795s | human constructs they fall prey to the |
| 1798s | same |
| 1799s | you know processes of Decay over time |
| 1802s | and they they're testing obsolete like |
| 1804s | the body system that I mentioned they're |
| 1805s | just testing obsolete functionality they |
| 1807s | also tend to |
| 1809s | be even become more difficult to read |
| 1813s | than the code itself so we've also put |
| 1815s | some effort into making the tests more |
| 1818s | uniform and easier to read so |
| 1821s | they're easier to work with |
| 1824s | um |
| 1825s | there so we worked on like this for a |
| 1829s | good stretch of time |
| 1831s | um eventually we came to a point where |
| 1833s | we were trying to move to a new platform |
| 1834s | we were making some new code paths they |
| 1837s | were parts of the system that we were |
| 1838s | you know planning to leave behind but we |
| 1840s | couldn't quite do it yet |
| 1841s | and so we it starts to become overly |
| 1844s | complicated to sort of |
| 1845s | maintain backwards compatibility while |
| 1847s | trying to go towards the future so at |
| 1850s | that point we came up with a new plan |
| 1854s | which was to make |
| 1858s | copies of the logic that we wanted to |
| 1860s | modify and take forward and so that we |
| 1864s | could leave the backwards compatibility |
| 1865s | in a separate copy of the same code and |
| 1868s | this is like the blue is the um the old |
| 1871s | bets where you see blue and purple is |
| 1874s | where we made copies and then the first |
| 1875s | place then you New pieces this is from |
| 1877s | an actual like design document or we |
| 1880s | were trying to like wrap our heads |
| 1882s | around what we were what we had on our |
| 1883s | hands we this decision was taken after |
| 1886s | careful deliberation it would have been |
| 1888s | a terrible idea to do this in the |
| 1890s | beginning because |
| 1892s | when we did this we had two copies of |
| 1893s | the same logic and it did mean in some |
| 1895s | cases we had to make changes the same |
| 1897s | change in two places so if we had |
| 1900s | started out doing this it would have |
| 1902s | been a lot of extra work and just a |
| 1904s | metal burden to keep track of what we |
| 1906s | were doing what we had done |
| 1909s | um but at that time we had made enough |
| 1911s | progress we knew the code well enough uh |
| 1915s | and we could see the Finish Line we knew |
| 1917s | this would be for a limited amount of |
| 1919s | time that we would have to do the |
| 1921s | duplicate changes |
| 1925s | um |
| 1926s | so this is where we are now |
| 1929s | our kitchen is a lot lot cleaner we |
| 1931s | still have this |
| 1933s | in the corner there which |
| 1936s | meaning that we still have one piece of |
| 1938s | functionality that still goes through |
| 1941s | the old bits the blue boxes there |
| 1944s | uh the replacement for that is actually |
| 1947s | in testing right now so we foresee we |
| 1949s | can cut this away within like a few |
| 1953s | weeks at most |
| 1955s | when that happens we can delete the last |
| 1957s | pieces of those blue boxes and at that |
| 1960s | point we will have deleted around 1 |
| 1961s | million lines of code so that will be a |
| 1963s | happy day |
| 1967s | and |
| 1969s | that means that includes simplifications |
| 1973s | like you saw before |
| 1975s | um maybe not very that there isn't a big |
| 1978s | chunk of the that number |
| 1980s | components that are no longer relevant |
| 1983s | and also like this Pierre Pere mentioned |
| 1986s | with Python 3 projects sometimes when we |
| 1988s | have newer Frameworks newer packages and |
| 1991s | so on we just we can do things in a |
| 1993s | simpler way we need less overhead we |
| 1995s | need less customizations and |
| 1996s | modifications |
| 1998s | um so it's uh a lot uh nicer easier and |
| 2003s | faster to work in that code now |
| 2007s | and for you the players that just very |
| 2009s | simply means we can do more and a better |
| 2011s | setup for the future |
| 2013s | we started out heading somewhere we had |
| 2017s | a rough idea how we're going to get |
| 2018s | there |
| 2020s | um this has been a journey we have taken |
| 2023s | many long conversations in the team it's |
| 2024s | very much been a team effort |
| 2027s | um deliberating water are we still |
| 2029s | heading in the right direction how can |
| 2031s | we and we've had to backtrack and all |
| 2033s | that |
| 2034s | but it's been very rewarding and |
| 2035s | especially now where we feel we're |
| 2038s | really starting to you know feel the |
| 2040s | benefits of this |
| 2043s | back to you Korea |
| 2046s | speaking of Journeys these upgrade |
| 2049s | projects are always Journeys and these |
| 2051s | Journeys look different depending upon |
| 2053s | the path that we have to take there are |
| 2055s | different risks that we have to deal |
| 2057s | with |
| 2058s | like ascending Everest you can try to go |
| 2060s | to straight line up or you can start |
| 2062s | here and then you realize oh you need to |
| 2064s | change path |
| 2065s | the Python 3 project was just one line |
| 2067s | of this well we just can go one way |
| 2070s | there's a lot of risk to get up there |
| 2073s | but we have to do it we need to prepare |
| 2075s | well so that we don't fail on our |
| 2078s | journey |
| 2079s | the payments team had a similar |
| 2081s | goal but they had the luxury to deviate |
| 2085s | a little bit on their path they could |
| 2086s | start here and then realize oh this path |
| 2088s | doesn't go forward let's try this one |
| 2090s | over there |
| 2091s | and that's how they achieved it |
| 2093s | so one of the first learnings from this |
| 2095s | upgrade projects for CCP is that |
| 2098s | different upgrade projects have |
| 2099s | different requirements and they may need |
| 2102s | to be approached slightly different |
| 2105s | but what else can we learn from these |
| 2106s | things |
| 2108s | one of the biggest learnings certainly |
| 2110s | and it's probably for every company in |
| 2112s | the world and it's obvious in hindsight |
| 2115s | always is that tribal knowledge will be |
| 2118s | a roadblock |
| 2119s | quite literally as in this picture |
| 2121s | maybe some of you know that picture when |
| 2123s | you Google for chesterton's fence |
| 2126s | this is one of the results that will |
| 2127s | come up what is chess settings fence |
| 2130s | that is essentially |
| 2132s | what Tribune the philosophical example |
| 2135s | for tribal knowledge |
| 2137s | the Sears fence that's going across the |
| 2140s | road |
| 2141s | it's in your way but can you remove the |
| 2143s | fence |
| 2144s | why is it there |
| 2146s | what's the purpose |
| 2149s | no one knows except for the people that |
| 2151s | put it up right there's no sign here |
| 2152s | that says like don't remove the fence or |
| 2154s | there's no note here that says like oh |
| 2157s | you know we put this fence up because |
| 2158s | the neighbors cheaper while moving |
| 2160s | freely and we need to keep them away |
| 2161s | from this part of the road |
| 2163s | nothing |
| 2164s | so it's a very very difficult situation |
| 2166s | to be in |
| 2167s | this is something we are facing a lot |
| 2170s | and we have taken measures internally to |
| 2173s | combat this tribal knowledge to have |
| 2175s | less of a roadblock when we do upgrade |
| 2177s | projects these things include code |
| 2180s | comments commit messages |
| 2182s | and architecture decision records |
| 2186s | they serve different purposes the commit |
| 2188s | messages and code comments are |
| 2190s | definitely very close to the code so |
| 2191s | they hopefully |
| 2193s | describe why the code was changed in a |
| 2195s | way nobody used to tell me this iterates |
| 2198s | 10 times over something rather tell me |
| 2200s | why you're iterating 10 times over this |
| 2202s | thing |
| 2202s | and in the case of architecture decision |
| 2204s | records it's like just saying like put |
| 2207s | up this fence because the neighbor's |
| 2208s | horses were feeding on my lawn |
| 2211s | another thing is that these upgrade |
| 2214s | projects we have to do is a great |
| 2217s | opportunity to pay down technical debt |
| 2219s | that we have |
| 2221s | you can never really justify technical |
| 2224s | debt projects in isolation there are |
| 2226s | many programmers which say like oh you |
| 2228s | know I would like to just spend two |
| 2229s | weeks sitting there fixing all these |
| 2232s | things in the design that I just like |
| 2234s | skimmed all over that when I was doing |
| 2235s | the the initial product but |
| 2239s | when you don't bring when you then bring |
| 2240s | this kind of suggestion to your |
| 2241s | production team it's like well but uh |
| 2244s | you know what kind of value are we |
| 2246s | producing those two weeks and it's |
| 2248s | really difficult to just say like well |
| 2250s | this is well your Force developers like |
| 2252s | there needs to be actually business |
| 2253s | value being created so that you know |
| 2256s | production will give you the buy-in for |
| 2258s | making this change that's a lot easier |
| 2261s | when you have a project that's critical |
| 2262s | like Python 3 update or the payment |
| 2264s | system update then you can say like Hey |
| 2267s | we're going to get this new payment |
| 2268s | system out of the door which gives us |
| 2270s | this benefit on top of it and at the |
| 2272s | same time we can make it easier for the |
| 2274s | future |
| 2278s | again |
| 2284s | yeah so |
| 2286s | the next learning is that it's really |
| 2289s | important to invest in observability and |
| 2291s | testing infrastructure for the software |
| 2294s | without observability we don't know |
| 2296s | what's going on at runtime you know we |
| 2298s | can have all our nice little demos and |
| 2301s | examples within our projects as they are |
| 2304s | but we don't know how it actually |
| 2305s | behaves on Tranquility you know then |
| 2307s | suddenly eight thousand of you jump into |
| 2309s | a system to fight each other it's like |
| 2310s | well what's actually happening you know |
| 2312s | like this was not supposed to happen |
| 2314s | like this but uh to be able to deal with |
| 2317s | those situations and to draw proper |
| 2318s | conclusions you need observability |
| 2321s | the other part is then the testing |
| 2323s | aspect where without a good test suit |
| 2326s | it's basically impossible to go ahead |
| 2329s | and exercise all the functionality in |
| 2332s | your project Eve is a super complex game |
| 2334s | as we all know and |
| 2336s | I think the regression test suit takes |
| 2338s | several days if run by a human |
| 2341s | so you don't want to spend that kind of |
| 2343s | time for every tiny change you make or |
| 2345s | for every upgrade project you make |
| 2346s | constantly so investing into the test |
| 2348s | coverage is really important there as |
| 2350s | well |
| 2353s | and it works again and um |
| 2357s | last but not least maintenance work is |
| 2359s | like tending your garden |
| 2361s | or cleaning the kitchen as we add |
| 2363s | furthermore but if you want to have a |
| 2365s | garden that's as beautiful as this one |
| 2367s | right everything neatly arranged |
| 2370s | cars evenly cut nice arrangement of |
| 2373s | flowers if you want something like that |
| 2374s | you need to put in a lot of work imagine |
| 2377s | you would leave this for like two |
| 2378s | seasons without any attendance that |
| 2380s | would be things growing all over right |
| 2383s | so it's a constant effort you have to |
| 2384s | put in |
| 2386s | if you don't put in the effort you don't |
| 2388s | get a good product that's as simple as |
| 2390s | it did as it is |
| 2392s | and |
| 2393s | with all these learnings this is how we |
| 2396s | tend to keep Eve running forever |
| 2398s | thank you |
| 2409s | I think we have just a few minutes for |
| 2412s | questions if you |
| 2414s | if you have any yes |
| 2421s | is there any kind of reality to that |
| 2424s | specific area |
| 2427s | can you repeat Mima |
| 2433s | yeah do you are you familiar with this |
| 2435s | Thomas |
| 2438s | uh the question is if there's any truth |
| 2441s | to memes about postcode |
| 2444s | I've I haven't been away yeah |
| 2449s | yeah I think I think that's best |
| 2451s | yes please |
| 2461s | so the question is about the no downtime |
| 2463s | experiment that we ran last year |
| 2468s | um |
| 2468s | there has been a little progress on that |
| 2471s | front |
| 2471s | since last year |
| 2473s | we are still aiming for it |
| 2475s | and we have picked off the easy |
| 2477s | low-hanging fruits |
| 2479s | but there are some things that take a |
| 2481s | bit more effort and they are currently |
| 2483s | being investigated |
| 2485s | it's similar to the Python 3 update |
| 2486s | right it's a very critical thing and uh |
| 2490s | so it takes very careful planning but |
| 2492s | it's definitely still on the table |
| 2510s | full in the essence of Dr |
| 2517s | documents so the question is about the |
| 2520s | availability of test servers and whether |
| 2523s | certain things are constantly available |
| 2524s | on them or not |
| 2527s | we are running many many different test |
| 2530s | servers intent internally and we do |
| 2533s | provide test servers for dedicated |
| 2535s | purposes |
| 2536s | like for example the native Mac client |
| 2539s | way before it was able to to be put into |
| 2542s | the main line where it would eventually |
| 2544s | be released we already wanted to gain |
| 2546s | confidence so we set up a test server |
| 2547s | environment where we connect with the |
| 2549s | native Mac client to that |
| 2551s | um so they may not always be available |
| 2554s | to the public |
| 2556s | but we definitely have a lot of those |
| 2558s | internally and we provide them as needed |
| 2562s | any more questions oh yes |
| 2572s | hmm |
| 2578s | you're making that process easier |
| 2581s | so the question is whether when we look |
| 2584s | back in 20 years and the next big major |
| 2587s | python upgrade comes along whether we |
| 2589s | would look back and say like hey yeah |
| 2590s | this is going to be easy or not |
| 2593s | um |
| 2595s | this is a really good question like it |
| 2597s | depends a lot on what will be happening |
| 2599s | in those 20 years right it depends a lot |
| 2601s | on the changes that that are coming |
| 2604s | one thing from the current learnings |
| 2606s | that we have and if that we have |
| 2608s | prepared for such future updates is that |
| 2610s | we are very diligent on how we keep |
| 2612s | track of the modifications we are making |
| 2614s | to python like with python 2 it's all in |
| 2617s | the python 2 code base itself littered |
| 2619s | over it as I said earlier like some of |
| 2621s | like most of it annotated not everything |
| 2623s | um but that makes it really difficult |
| 2625s | that when you just take the new version |
| 2627s | any new version of python really right |
| 2629s | we were on 2.7.1 if it would go to 2.7.2 |
| 2632s | already it's really difficult to judge |
| 2634s | what is a genuine change in Python and |
| 2637s | what is the thing that we modify that |
| 2639s | needs to be adapted we have found a |
| 2641s | solution for that internally where we |
| 2642s | basically just get the normal sources of |
| 2645s | stackless python and then just have a |
| 2646s | set of patch files |
| 2648s | all individually containing |
| 2650s | functionality that we need that then |
| 2652s | gets successfully applied on the |
| 2653s | original sources so when we then take a |
| 2656s | new version of python and there is a |
| 2658s | change that affects our |
| 2660s | our modifications then that specific |
| 2662s | patch is going to fail to apply and we |
| 2664s | know exactly what we have to look at so |
| 2666s | it should definitely make a lot of |
| 2668s | things easier going forward |
| 2671s | and maybe then to pick up from that on a |
| 2674s | slightly different note or like how what |
| 2677s | we think will happen in the next 20 |
| 2678s | years and where we will be then so all |
| 2681s | our efforts now you may have heard about |
| 2682s | the quaser platform so we've very much |
| 2686s | working towards what we think will help |
| 2688s | us to going forward to reduce these |
| 2691s | reduce the scope of these big changes uh |
| 2694s | making when we make additions to make |
| 2696s | them in in isolated services that are |
| 2699s | more manageable as units rather than and |
| 2702s | this is as you know how uh where the |
| 2706s | trend has been going in the recent years |
| 2708s | from the big monoliths that we used to |
| 2712s | build to use little components that are |
| 2715s | more controllable also like Podium |
| 2718s | mentioned earlier the |
| 2719s | observability and automated tests and |
| 2722s | these things that sort of help us to |
| 2723s | keep things in a good shape and make it |
| 2725s | easier to |
| 2727s | take them forward there's a question |
| 2730s | that's the last question that we can |
| 2732s | take sorry yeah |
| 2764s | so I think the question was like do we |
| 2766s | have |
| 2773s | yes I mean we have I mean the company |
| 2777s | and all the development practices are |
| 2779s | obvious they have like learned a lot and |
| 2781s | matured a lot over the years the |
| 2783s | question is are we documenting better |
| 2785s | what we do and I think a previous |
| 2786s | example of of how we commit uh how we |
| 2789s | document our commit is a very good |
| 2791s | example this was one of the first things |
| 2793s | that I learned when I joined CCP |
| 2796s | was when you commit like just |
| 2799s | short description link to the work item |
| 2801s | and why are you doing it don't tell me |
| 2804s | you know at that |
| 2806s | I don't know other than if statement |
| 2809s | I can see that that's not helpful why |
| 2811s | did you do it |
| 2812s | and just the paper trail like link to |
| 2816s | the work item so that and this happens |
| 2818s | we uh sometimes go back you know 10 |
| 2821s | years wide was this change made so we |
| 2824s | have learned a lot we're doing a lot |
| 2826s | better definitely from the early days |
| 2828s | and if you were sort of startup you also |
| 2829s | know like it's just different |
| 2831s | requirements when you're getting started |
| 2833s | and you just want to get something out |
| 2835s | the door you that's a different mindset |
| 2838s | in your in a different |
| 2840s | sort of there are different things that |
| 2842s | are more important than and then you |
| 2845s | need to settle into sort of a more |
| 2847s | disciplined production Cadence and |
| 2850s | that's definitely for like from having |
| 2852s | been with CTP since 2007 definitely see |
| 2855s | a lot of |
| 2856s | uh |
| 2858s | maturity and in that |
| 2861s | all right thank you very much for |
| 2863s | attending |