WEBVTT

1
00:00:02.520 --> 00:00:12.960
William Cheng: Welcome to the first part of lecture 25 so we're in Colonel three now Colonel three is you have little over two and a half week to finish.

2
00:00:13.679 --> 00:00:21.420
William Cheng: If you have co from previous semester, don't look at them. Don't copy them as to get rid of it. You just need to start early. If you want to have a chance of finishing Colonel three

3
00:00:21.690 --> 00:00:32.340
William Cheng: And again, Colonel three is bigger than Colonel one and two combined. So, you know, you need to start early grading Ghanaians. The only way will grade and you know for Colonel three don't guess

4
00:00:33.180 --> 00:00:41.430
William Cheng: I guess just like all the colonel assignment don't get stuck too long. So one thing I recommend people to do. I think one student is doing that nobody else is doing that.

5
00:00:42.090 --> 00:00:50.460
William Cheng: Maybe nobody else has get to the first page fall yet where you get you get your first legitimate page fall that's in the function call handle page fall

6
00:00:51.030 --> 00:00:54.600
William Cheng: Post your print out of this GDP command, it will print out your address space.

7
00:00:55.530 --> 00:01:04.380
William Cheng: So you posted in the class Google group and acid. Other are seeing the same thing. Okay, so please again, don't just as other people to post you post your stuff and ask other people to comment.

8
00:01:04.710 --> 00:01:13.530
William Cheng: On your data structure, right. So I guess one comment I give is that, you know, I guess all the address space that we draw, you know, the I just basis sort at

9
00:01:14.010 --> 00:01:27.990
William Cheng: A small address to large address. And I think that student. The, the, the order is reversed. I don't know if it matters, but you know, I think it's probably better to be sorted in the right order. So anyways, so. So that's one thing I noticed, man.

10
00:01:29.610 --> 00:01:30.930
William Cheng: So you

11
00:01:32.370 --> 00:01:39.690
William Cheng: If we start getting to see a lot of messages and you want me to answer a particular question. You have to send me a private email. Okay.

12
00:01:40.080 --> 00:01:47.010
William Cheng: You know, so when you send me a private email. Again, you know, I cannot tell you what code. Right, right. So you're going to ask me what code, right. I can't tell you so.

13
00:01:47.550 --> 00:01:51.390
William Cheng: That would be the worst case scenario, otherwise I try to sort of point you in the right direction.

14
00:01:51.810 --> 00:01:55.470
William Cheng: Okay. And if you, you know, and also, you know, I do not have to

15
00:01:55.800 --> 00:02:09.330
William Cheng: You know, respond to question posted in the class guru, even if you address the question directly to me. And the reason for that is that intellectual one I made it very, very clear if you send me a private email, I guarantee to respond to you within 2425 hours.

16
00:02:09.870 --> 00:02:13.500
William Cheng: Okay, if you don't send me a private email. I didn't make any promises.

17
00:02:14.010 --> 00:02:22.740
William Cheng: Okay, so, so if you want to, you know, post into the class Google Groups. I do. Professor this in that that's not a private message. Okay. You have to do it properly. Okay, if you want me to answer right.

18
00:02:23.490 --> 00:02:37.800
William Cheng: So, so again, there's a lot of people posting a lot of stuff. So I won't be able to comment on everything. Okay. So as long as there's a sort of a discussion going on. And if you if you're totally confused about the discussion, feel free to send me a private email. Okay.

19
00:02:39.360 --> 00:02:51.420
William Cheng: All right, your recommended Colonel three timeline is by now P frame Darcy with VM equal to zero, F, F, F is equal to one should be working. So again, that's your assignment supposed to be doing, you know, you know,

20
00:02:52.020 --> 00:03:02.820
William Cheng: In the past weekend. You also need to make sure BFS run perfectly with system for file system equal to one. So again, you might run into bugs. Right. Some people already find some really weird bugs that

21
00:03:03.690 --> 00:03:16.410
William Cheng: So So anyways, so you should fix all your bags and then proceed. So once you can run BFS test successfully with a system of us is that you sent via email and put one and now you're working or user space programs.

22
00:03:16.860 --> 00:03:28.170
William Cheng: Okay, so, so the timeline is that by the end of this weekend. So this is the first week of kernel three, you should build the address space and run the hello program. And, you know, again, don't follow the

23
00:03:29.340 --> 00:03:38.820
William Cheng: The, the Phoenix documentation. Right. They are being too aggressive. So my recommendation is to run the only this way. Okay. So if I'm in a prog run right. And probably you don't need the acacia anymore.

24
00:03:39.000 --> 00:03:44.070
William Cheng: All you need to do is to call it, Colonel. Exactly. And the first argument is the hello program.

25
00:03:44.520 --> 00:03:56.940
William Cheng: Okay, so this way you're colonel in a process will go into the user space become the user space in that process, but in this case the user stay in the process was simply print hello and then it will come back inside a kernel and then you turn off the machine.

26
00:03:58.200 --> 00:04:04.350
William Cheng: Okay, so, so that's all you have to do during this first week, right, if you can get that to be done. That's great. It's a lot of work.

27
00:04:04.980 --> 00:04:12.090
William Cheng: So this is not easy. And a lot of work. OK. So again, look at the discussion section a video so you know

28
00:04:12.660 --> 00:04:19.410
William Cheng: Two weeks ago and I show you what function has to be implemented. There's a lot of function has to be implemented some of these function know a little weird.

29
00:04:19.710 --> 00:04:24.090
William Cheng: Okay. And that can be a little tricky. So again, you know, if you're stuck, you know, send me email now.

30
00:04:24.900 --> 00:04:31.650
William Cheng: Alright, so after you get hello at work during the week two of the colonel three assignment that will be next week.

31
00:04:31.920 --> 00:04:37.500
William Cheng: Get all the basic user space program to run directly using exactly the same way around one program at a time.

32
00:04:37.680 --> 00:04:47.760
William Cheng: Once you get one program around, you know, go to the next one. Go to the network. I mean, if you really want to write a show commands. Each command will run a different users Facebook. That will be fine, too. Okay. But again,

33
00:04:48.660 --> 00:04:51.000
William Cheng: You know that that might be a little more difficult.

34
00:04:51.780 --> 00:05:00.210
William Cheng: So doing the most straightforward way is simply your kernel in a process become the user space in a process to run the command to run the program.

35
00:05:00.420 --> 00:05:05.280
William Cheng: That are specified in the grading island and they come back into the kernel and then you shut your turn on the machine now.

36
00:05:05.670 --> 00:05:15.720
William Cheng: Alright, the last test over here is it's called for. Can't wait. Right. So that will be the first user space program that you saw at the beginning of the semester, the parent for code for child wait for the child to die.

37
00:05:16.050 --> 00:05:18.570
William Cheng: Except that this call also do copy on write

38
00:05:19.050 --> 00:05:25.200
William Cheng: Okay, so, so the point over here for the last with what this program is to make sure that you see, you get your shadow object to work correctly.

39
00:05:25.470 --> 00:05:32.550
William Cheng: Okay, before you run this program, you don't shout object. Right. But in order for this one to run. If you want to perform copyright correctly, you have to use a shadow object.

40
00:05:32.970 --> 00:05:35.550
William Cheng: That. All right. So by the end of the week to

41
00:05:35.910 --> 00:05:45.450
William Cheng: You know, after you get that to work. The next thing to do is to, you know, do more fork. Right. So in this case, you're gonna run ESPN in it. That's the real user space in a program. So again, you run it exactly the same way.

42
00:05:46.050 --> 00:05:56.880
William Cheng: That. So if this program works successfully. What it will do is it will create a child process and retail process is going to be a user space shell, which is known as a slash bins last sh.

43
00:05:57.210 --> 00:06:03.750
William Cheng: Okay that's users, Michelle, and that's a regular Unix problem, it will ask you, you know, to type in a command and they will run your program and stuff like that.

44
00:06:04.920 --> 00:06:12.210
William Cheng: Okay, so, so, so, so again, and the spin in it. This will be the first time you call Malik, and you're going to run into bugs.

45
00:06:12.540 --> 00:06:15.780
William Cheng: Okay, because he didn't really implemented correctly. So, you know, because

46
00:06:16.140 --> 00:06:24.990
William Cheng: Because you never have to worry about it. So now, will you get their chances are you gonna run into a bar, you have to fix them. You know, again, you have to make the guess it makes the end map system call

47
00:06:25.560 --> 00:06:34.320
William Cheng: So you need to write code for you guys. So, therefore, you know, towards the end over here, there's still some additional work to do in order to get to get the SP network.

48
00:06:35.190 --> 00:06:44.100
William Cheng: That. So once you get has been in the world, you should test out all your other program to make sure that you can run those programs directly from the user level shout as you should be able to say hello.

49
00:06:44.280 --> 00:06:52.080
William Cheng: You system as a, you know, our IDs and all the other programs over here in the basics. As you should just be able to type in a command line, and then they should work.

50
00:06:52.770 --> 00:06:59.610
William Cheng: Okay. So once that's done, then the week three of the colonel three, you need to pass all the tests and a great guy. Right. So again, there are five major test.

51
00:07:00.270 --> 00:07:08.340
William Cheng: You know, again, look at the waiting guideline, you have to run BFS. Again, there's something called fork bomb. There's mentality. Eat man there's a stress test.

52
00:07:08.970 --> 00:07:17.700
William Cheng: So, you know, again, you have more functions at the right you have more testing that you have to do so. So that yeah so so again this this timeline is very, very aggressive

53
00:07:18.240 --> 00:07:27.180
William Cheng: Okay, if you want to make sure that you get, you know, you finished Colonel three all the way through. Then, my recommendation is to follow this aggressive time, you know, time, ah,

54
00:07:27.900 --> 00:07:35.370
William Cheng: OK. So again, you can actually get quite a bit of partial credit, even if you don't finish all the way to the all the way to the end. Okay.

55
00:07:37.230 --> 00:07:47.610
William Cheng: All right. And again, it's important that you read the colonel three FAQ and understand pretty much all the lecture material cover so far in the class. Okay, the spec of the Phoenix documentation.

56
00:07:47.820 --> 00:08:01.860
William Cheng: Is not enough, you have to look at the FAQ, you have to, you know, think about everything that you have learned, because a lot of the stuff that we went over in lectures, they are getting you ready for Colonel three. Yeah. Alright. So again, you know, ask me question if your stuff.

57
00:08:06.150 --> 00:08:12.960
William Cheng: Alright, so we're going to continue to talk about crushing resiliency. So last time we finished talking about software update

58
00:08:13.440 --> 00:08:16.980
William Cheng: So today we're going to talk about the second method which is running transactions. Yeah.

59
00:08:17.580 --> 00:08:24.060
William Cheng: So, so a transaction is a database concept. So we're going to use a good old database you know example to explain transaction.

60
00:08:24.570 --> 00:08:29.460
William Cheng: The classic example is to transfer money right want to transfer $100

61
00:08:30.180 --> 00:08:38.010
William Cheng: There from account to account number two is over here. Here's a condom one. So let's say account number one you have $900 right and then count number to over here.

62
00:08:38.430 --> 00:08:46.260
William Cheng: You only have you know 100 $100 something like that, you know, you need to write a $200 check right. The second the account to over here the checking account.

63
00:08:46.530 --> 00:08:53.610
William Cheng: And account number one over here is your savings account. So what you need to do is I need to transfer $100 from accountable one to account number two.

64
00:08:54.600 --> 00:09:02.700
William Cheng: Okay, so how do you do this. Wow. So you know there's there's really no atomic operation that will actually

65
00:09:03.210 --> 00:09:06.480
William Cheng: Allow you to do this you do this in one shot.

66
00:09:06.870 --> 00:09:13.200
William Cheng: So in a way, this is very similar to the move command that you have to implement Colonel to right. How do you move a file from one

67
00:09:13.410 --> 00:09:21.210
William Cheng: Director to another directory right you unlinked from the first directory and then you link and into the second directory. Well guys, this has been done in two steps.

68
00:09:21.630 --> 00:09:32.520
William Cheng: Now, so, therefore, the way we're going to do is that we are detrimental number one by $100 and then increment account number two by $100 and then we're going to lock these two operations together in one atomic operation.

69
00:09:33.030 --> 00:09:39.990
William Cheng: Okay, so, so, so the way this is done right, is that I'm going to create a transaction. Okay, so what is the transaction transaction. It basically it's a contract.

70
00:09:40.470 --> 00:09:47.160
William Cheng: Is a contract that you signed with the databases them to say that these two operations over here. I'm going to either do all of them are going to do. None of them.

71
00:09:48.300 --> 00:09:58.590
William Cheng: Okay, so why because if I do, none of them, or if I do, none of them when this operation is done. He was a border. I'm going to end up. I still have $1,000 in all my accounts or I might, you know, if I write a check by check will bounce.

72
00:09:58.950 --> 00:10:02.220
William Cheng: But at least I didn't really lose any money. I said $1,000 okay

73
00:10:02.610 --> 00:10:14.400
William Cheng: If it turns out that this you know this transaction was executed successfully. Then I'm going to end up with a ones that I have $100 in them. And then he was going to have $200 in that. And now when I write a check. I won't bounce and also the

74
00:10:14.730 --> 00:10:24.060
William Cheng: Total amount is that you've used to account is still $1,000 okay so to try it. So, so one of the ways you know, think about is that a transaction either done all or nothing.

75
00:10:24.840 --> 00:10:36.960
William Cheng: Okay, so that's what part of the semantics of this kind of a strange contract, I'm typically when you sign a contract, you say, I'm going to do it. Okay, so this is the weird contract user that either you don't do any of it or you do all of it, but you can do nothing in between.

76
00:10:38.460 --> 00:10:45.780
William Cheng: All right, I'll be here. It says, also, you know, you also need to run a transaction was satisfying something called the acid property acid is the acronym.

77
00:10:46.620 --> 00:10:56.850
William Cheng: A SENSE FOR THE FOLLOWING A stands for atomic right so therefore either all or nothing. So, so there's no way for you to do part of the transaction either do all of them and do none of that. Yeah.

78
00:10:57.540 --> 00:11:08.160
William Cheng: So how do you know whether you're doing all of it, or none of them. Okay, so there's a concept called a commitment time that's important concept so horizontal axis over here, this is time, right, there's some time over here. No, as the commitment time

79
00:11:08.700 --> 00:11:13.860
William Cheng: Okay, so, so that the commitment time determine whether you do it all of it or you do not have it.

80
00:11:15.150 --> 00:11:26.700
William Cheng: Okay, so, so once you start running stuff inside of transaction. Okay. And I want some point, you got to come into the church and transaction if you come if you get to commit your transaction that you're making a promise that all of them will be

81
00:11:27.210 --> 00:11:33.090
William Cheng: All of them will be done. Okay. But for some reason, you get a system crash before you commit to the transaction, then none of it will be done.

82
00:11:33.780 --> 00:11:44.730
William Cheng: Okay, so sometimes we need to define exactly when this commitment time is because it's coming at a time is very, very important, right, if you get a crash before that. Well, then nothing will be done if you get a crash after that is guarantee that the transaction will be done.

83
00:11:45.510 --> 00:11:52.050
William Cheng: Okay. All right. The C stands for consistent. So, so again, these, these terms are coming from the database, you know, community.

84
00:11:52.410 --> 00:12:01.530
William Cheng: Sequences that it will take the file system. And so again, in this case, we're going to apply into the file system and not a database of them it will take the file system from one consistent state to another.

85
00:12:02.220 --> 00:12:07.350
William Cheng: Okay. So, therefore I'm going to assume that before you start running any transaction your file system is inconsistent state.

86
00:12:07.680 --> 00:12:12.450
William Cheng: Okay, so now what we try to run one transaction on top of it. It will take the file system to another trans

87
00:12:13.230 --> 00:12:25.620
William Cheng: American systems day and if you run another transaction, it will still go into a transaction state. Okay, so. So one thing that you need to understand is that once we start running transactions. If you want to modify the file system, you have to use transactions.

88
00:12:26.040 --> 00:12:32.310
William Cheng: There's no way for you to modify the file system without a transaction anymore. Obviously if you if you tried to modify the file system just any way you want.

89
00:12:32.550 --> 00:12:40.200
William Cheng: You're going to end up you're going to take the classes. I mean to inconsistency day once they're in inside inconsistent. Stay well, then all these guarantees are useless.

90
00:12:41.190 --> 00:12:44.520
William Cheng: Okay, so once you start running any transactions, how to process them.

91
00:12:44.730 --> 00:12:55.470
William Cheng: Anytime you want to change the fosters them you have to use a transaction, you have to sign the contract, you got to put what you need to do instead of transaction. And then if houses them say, I'm going to go run the transaction to make sure this is all of this done, or none of it's done.

92
00:12:56.850 --> 00:13:04.710
William Cheng: Okay, so, so again, when you try to construct this. The other this contract, you need to write it out, you know, you need to write the stuff that you want to do very, very carefully.

93
00:13:04.890 --> 00:13:14.070
William Cheng: Because you need to assume that before you read this transaction, the system is inconsistent state. The end after you put on the transaction, the system will also be in the transact consistency.

94
00:13:14.730 --> 00:13:23.880
William Cheng: So what is the consistency. Right. I mean, in my previous example, the consistency state is that if I add my two accounts, you know, balance there will be equal to 1000 $1,000

95
00:13:24.570 --> 00:13:31.320
William Cheng: Okay, so that's, for me that's considered consistent. Okay, so once you start running transaction, it will always take the system, you know,

96
00:13:32.790 --> 00:13:35.370
William Cheng: From one consistent state to another state.

97
00:13:39.270 --> 00:13:44.970
William Cheng: Right. The I stands for isolated. It says that it has no effect on other transaction until it's committed

98
00:13:45.450 --> 00:13:56.190
William Cheng: So again, the commitment hamburger important once you commit a particular transaction. Now you should assume that the system going to the new state. So if you want to run other transaction, they will be based on the new state.

99
00:13:57.060 --> 00:14:05.550
William Cheng: Okay, if this, you know, so, so, so, so, so if you started transaction. If you haven't committed are you allowed to run other transaction that's based on the current state variable.

100
00:14:05.790 --> 00:14:13.050
William Cheng: One. So, so by by this definition over here that you are not allowed to run other transaction that are based on the changes that you're making inside your transaction.

101
00:14:13.740 --> 00:14:23.670
William Cheng: Okay, so therefore only once you have committed transaction. Then you allow other other transaction to to go forward, based on you know the the the system state that

102
00:14:24.120 --> 00:14:28.500
William Cheng: That you will present. Well guys, again. Will you try to write. Right. So it's actually that'd be very, very careful.

103
00:14:28.890 --> 00:14:33.930
William Cheng: We're not going to go into too much detail of exactly how to write a transaction, right, because when you try to modify the file system.

104
00:14:34.140 --> 00:14:42.330
William Cheng: You might modify, you know, many, many things at the same time. Well, does it follow all these rules. So again, you need to be very careful. Right. So that part is beyond the scope of this class.

105
00:14:43.200 --> 00:14:55.830
William Cheng: Okay, so over here was still sort of just want to show you, you know how to actually run a transaction inside the opera inside of our system that alright. The last one is durable so durable stands, you know, I guess people sort of try to come up with these

106
00:14:56.940 --> 00:15:09.720
William Cheng: These words just so that they can say acid. So these dense was durable durable, they mean persistent. Right. So why did you create this transaction over here. So the transaction is a contract the contract can just disappear.

107
00:15:10.230 --> 00:15:19.710
William Cheng: Okay, so therefore, where do we store this contract. We need to store this contract in a storage device so that in case we have a system crash. We still have a contract, right. So, especially after commitment time

108
00:15:19.890 --> 00:15:29.640
William Cheng: Once you have committed through transaction, you have a system crash, you need to remember what the transaction was okay. So, therefore, again, it needs to stay around, even after a system crash Yeah.

109
00:15:30.810 --> 00:15:35.580
William Cheng: Alright, so again, the important thing over here is that once you start running transactions to modify the file system.

110
00:15:35.820 --> 00:15:43.500
William Cheng: The only way to modify the file system is to run transactions. You are not allowed to modify any part of the process then without using a transaction.

111
00:15:43.980 --> 00:15:50.070
William Cheng: Okay. And, you know, running a transaction is going to slow things down because it's a little more involved. Okay. So, therefore, you know, this

112
00:15:50.370 --> 00:15:57.600
William Cheng: We saw before. In the picture right you know you know on the horizontal axis is consistency on the vertical axis is performance, right.

113
00:15:58.140 --> 00:16:08.340
William Cheng: Soft update is right here. But when you start running transaction is going to be right here. It's more consistent, but the system performance is going to be a little worse because you know there's no free lunch. Yeah.

114
00:16:10.950 --> 00:16:17.490
William Cheng: All right, so let's say, you know, let's see, how do you actually run a transaction. There are two different approaches. One is called journaling.

115
00:16:18.270 --> 00:16:26.430
William Cheng: So what's the journal, write your notes like a diary. Right. So would you do that, you know, so what is special about a journal right so journal. It's like, it's like a log. Right.

116
00:16:26.760 --> 00:16:32.400
William Cheng: It's a pen, only it's never modify. I mean, you're really not supposed to modify your journal, right, because it just requires sort of like a

117
00:16:32.820 --> 00:16:38.490
William Cheng: Sort of a history of what happened, right. So, therefore, you know, so, so, so over here. We're going to sort of made the connection between generally

118
00:16:38.700 --> 00:16:49.050
William Cheng: And the log structured analysis and all we mentioned before the launch of your file system is not very, very useful. But the idea that was developed in the last retro file system. Now they can be used inside journaling.

119
00:16:50.130 --> 00:17:00.270
William Cheng: Okay, so that's why it was important for us to talk about the locks or two of our system because because when you use those techniques to implement journaling. Okay. The other approaches shadow page. And we're going to talk about shadow page.

120
00:17:01.890 --> 00:17:07.470
William Cheng: After we finished talking about your own. So now the first one talk. But generally, the idea here is that before we update the desk.

121
00:17:08.190 --> 00:17:13.890
William Cheng: Okay, so what we need to do is that we need to we need to write things out into the journal. What we are going to do.

122
00:17:14.310 --> 00:17:21.150
William Cheng: Okay, so there's two kind of generally one is called undo journal and the other one is called we do journaling. It depends on what you need to write into the journal

123
00:17:21.570 --> 00:17:33.090
William Cheng: Okay, so if you write that before images of the data you try to modify into the journal. This is called undo journaling. Okay. Because later on when you have a crash, you can actually go back to the previous consistent state.

124
00:17:33.750 --> 00:17:39.840
William Cheng: Rather, the idea over here is that is that whatever you going to modify you save a copy of the previous state into the journal

125
00:17:40.470 --> 00:17:46.560
William Cheng: So this way when you crash. You can go back to the previous day, as if nothing has done as if nothing was nothing was done.

126
00:17:47.040 --> 00:17:52.380
William Cheng: That's it. Again, you know, so, so, so, so, so this is what this is how you go back in the previous day.

127
00:17:52.830 --> 00:17:59.040
William Cheng: Okay. In database community. This is no as roll back. Okay. But over here Fox's them. We're going to call it undo generally get

128
00:17:59.550 --> 00:18:07.260
William Cheng: The other one is called we do generally man. So in this case, we're going to write the after images of this blog everything through journal. Right. So when we finished right you know

129
00:18:07.950 --> 00:18:17.370
William Cheng: Running the transaction. We're going to take the system into the new state right in the new state several this blog inside of us. That will be modify. We're going to take the new image or we're going to store them into a journal

130
00:18:17.940 --> 00:18:25.500
William Cheng: Okay, so this way after a crash, we can actually go to the journal copy all those this bump into our file system. And now we can go into the new state.

131
00:18:26.550 --> 00:18:36.450
William Cheng: Or so in the databases and they actually use both techniques. They have a way to rollback. They also have a way to roll forward, typically in the file system, we see a reader generally more more

132
00:18:37.350 --> 00:18:51.060
William Cheng: Often okay so therefore we not going to talk about a new journey we're only going to talk about regionally. So in this case, we're going to write the after images of the this blog into the journal and this way. When we get a crash, we can actually take the system into the new state. Yeah.

133
00:18:53.340 --> 00:19:03.030
William Cheng: All right, so, so, so where's this journal write a journal is not part of the file system. The journal is a separate part of the desk. So if you think about a disk over here. So what's on the desk.

134
00:19:03.270 --> 00:19:09.900
William Cheng: On the desk. There's a swap space right the spa space right here. Okay, there's the actual file system. Right. So again, what's inside the actual process then

135
00:19:10.200 --> 00:19:15.480
William Cheng: There's a file system hierarchy. There's a free blog and now there's another part of that this is a journal in it.

136
00:19:16.350 --> 00:19:26.220
William Cheng: OK. So again, the characteristic of the journal is that journal is a pen only and never modify just like a log structure file system, but the journal is not a lot structure of our system.

137
00:19:27.390 --> 00:19:33.690
William Cheng: OK, so the journal looks like a long structure versus them because it's a pen only and never modified, but it's not a lot bunch of us. Let me just a journal

138
00:19:34.020 --> 00:19:42.510
William Cheng: Guys, we're going to see how to actually use a journal. The journal has one more operation that's not permitted on a rock lot for devices. And so this is to actually clear the journal

139
00:19:43.560 --> 00:19:44.550
William Cheng: Okay, this space on the

140
00:19:45.030 --> 00:19:54.810
William Cheng: On the desk over here that's reserved for journal. At some point, and actually clear out the entire Journal, which is not available for the larger file system right the logs read your file system. He keep adding stuff to it.

141
00:19:54.990 --> 00:20:03.540
William Cheng: In the end, it become a read only file system. So it's kind of useless. Okay, but the journal is very useful because at some point can actually leave the entire journal and now you can start a new journal

142
00:20:04.830 --> 00:20:10.260
William Cheng: Okay, so, so, so this, you know. So again, it's a little different from the law charge of us. Yeah. What's important over here is that

143
00:20:10.680 --> 00:20:16.230
William Cheng: You know, you can add journaling to any file system. Right. So if you have an actual process. And that's not crushed resilient.

144
00:20:16.710 --> 00:20:25.860
William Cheng: All you have to do so at journaling to the you know the the actual file system and now you're actually assist them can become a crushers this be become resilient.

145
00:20:26.730 --> 00:20:33.360
William Cheng: Okay, so that's why people love to run transactions, because whatever system you have you can make a crush Brazilian by adding journaling to it.

146
00:20:34.410 --> 00:20:40.620
William Cheng: Okay, so, so, so this is, you know, this is why everybody's using journaling today, you know, because you know he does that reason.

147
00:20:41.370 --> 00:20:50.100
William Cheng: Okay, so a journal is a pen only like a lot right you know and you know it's also never modified, but you haven't additional operation to clear the entire journal. Yeah.

148
00:20:50.310 --> 00:20:57.030
William Cheng: So we're only going to talk about a review journal with the review journal. We're going to append to what you are going to write to the main part of the desk.

149
00:20:57.240 --> 00:21:03.930
William Cheng: Which is the actual file system. Right. So we had this is the main part of the desert. There's a swap. There's a journal. The main part of our system is the actual file system.

150
00:21:04.260 --> 00:21:13.170
William Cheng: Okay, whatever. This blah, you're gonna you're going to end up writing to the actual process that before you modify the actual file system. The first thing you need to do is that you need to write it to the journal

151
00:21:13.650 --> 00:21:21.150
William Cheng: Okay, so here's a journal. We, here we saw before, we're modifying x, y, and z, x, y, and z or dis blocks. Okay. So, therefore, before we go.

152
00:21:22.350 --> 00:21:27.900
William Cheng: Before we modify the actual process that we're going to write them into a journal. So this is the new as is a new ideas and new z.

153
00:21:28.350 --> 00:21:33.390
William Cheng: This is what we're going to look like after the, you know, after the file system has been updated that

154
00:21:34.200 --> 00:21:41.820
William Cheng: Right, so, so we're doing this because we're doing a review journal right we're doing undo journal that we need to record the previous state of x, y AMP z.

155
00:21:42.480 --> 00:21:51.420
William Cheng: Okay, so in the fastest and through the Lucy and redo journal. So therefore, we're going to write the new state x y AMP z into the journal first and then we're going to append a commit record.

156
00:21:52.230 --> 00:21:57.990
William Cheng: Okay. So coming records a special record in the journal, you can actually look at the pattern going to be able to tell that this is coming record.

157
00:21:58.530 --> 00:22:06.480
William Cheng: It's one display concise. Okay, so that this will guarantee you gotta come in record is either written to the desk or not. And nothing in between.

158
00:22:07.260 --> 00:22:14.610
William Cheng: Okay, so therefore, in the journal. We are going to write x, y, and z. And then we're going to run a company record, we can actually verify that the coming record went on to the journal

159
00:22:15.600 --> 00:22:18.450
William Cheng: Okay, so what if you get a crash in between, right, if you get a crash in between.

160
00:22:18.870 --> 00:22:29.250
William Cheng: So for those of you who are a little bit familiar with it is that this has some kind of error correction code or error detection code if one bit is wrong inside this coming record, then he will say that the common records not on the desk.

161
00:22:30.270 --> 00:22:35.040
William Cheng: Okay, so therefore this company record. It's either in the journal or it's not in general. Okay, there's nothing in between.

162
00:22:35.670 --> 00:22:44.520
William Cheng: Okay, so therefore once we write the coming record into the journal. We can verify the the the camera has gone on to the journal. At that point we have committed to this transaction.

163
00:22:45.180 --> 00:22:51.240
William Cheng: Okay, so what is this transaction right this transaction is to take x, y, and z from the state to the mistake.

164
00:22:52.050 --> 00:23:04.860
William Cheng: Okay, so this is a commandment time once we ride to come in record into the journal and we verify that it's there. Now we have made a commitment to say that this transaction will be done. Okay, no matter how many times we crash it will always be done. Yeah.

165
00:23:06.990 --> 00:23:15.150
William Cheng: Alright, so, so, so what happened is that you know we we saw before, you know, you're going to modify the you know the the

166
00:23:15.660 --> 00:23:18.030
William Cheng: This update tasks. What it will do is that it will

167
00:23:18.570 --> 00:23:23.760
William Cheng: You know, look at all the dirty Bloss gather them all together and then read them out to this one at a time in some kind of a

168
00:23:24.030 --> 00:23:27.270
William Cheng: Strange order. However, the order this update has decided to do

169
00:23:27.510 --> 00:23:35.100
William Cheng: Okay, so now it's no longer like that, right, because all these x, y, and z over here is to put inside of transaction. So you should think about the transactions that data structure.

170
00:23:35.280 --> 00:23:41.160
William Cheng: Inside this data structure is going to contain all these 30 blocks and the dirty blogging is going to be the new state of the file system.

171
00:23:42.090 --> 00:23:48.840
William Cheng: Okay. So, therefore, what you know what the actual person has to do over here. So again, the actual process. And this is the one with journaling.

172
00:23:49.170 --> 00:23:57.930
William Cheng: What it will do is that, you know, once in a while, it will look it will try to find all these transaction that hasn't been committed to the desk and what they will do is that they will write x y AMP Z on

173
00:23:58.200 --> 00:24:04.980
William Cheng: The open up the transaction, look at the new state and invite it into the journal and they were to come in record and then they will go to another transaction.

174
00:24:05.310 --> 00:24:18.090
William Cheng: That. So once you rather come in record. What do you will do is it will release x, y, and z to the disk update tasks and now that this update has can write x, y, and z into the actual forces them in any order at anytime he wants.

175
00:24:18.840 --> 00:24:24.480
William Cheng: Okay, so that's what I mean by release it to the does have a task, but it does happen, it has can do whatever he wants right in any water they

176
00:24:24.900 --> 00:24:38.010
William Cheng: Probably run a strange algorithm or complete different algorithm doesn't really matter. I will release x, y, and z to the district that has that go ahead and modify anytime you want. Because I guarantee you that no matter when you crash. I will make sure that the transaction is completed.

177
00:24:39.420 --> 00:24:47.190
William Cheng: Okay. Alright, so once I do that. So again, I'm going to clear out this picture a little bit, right, I'm going to write x y AMP z over here within a transaction into the journal

178
00:24:47.460 --> 00:24:58.200
William Cheng: Write a company record verify the coming record is good, then I will release X, Y, and Z to that this update has I will go to the next transaction in the next transaction is a, b, c, d, right. So as the ABC and D.

179
00:24:58.410 --> 00:25:06.510
William Cheng: I will write a, b, c, d into the journal and this one is the afternoon edge and then I will write a coming record over here, I will verify that they went on to the

180
00:25:06.870 --> 00:25:20.070
William Cheng: To the journal and then I will release a, b, c, d into this update has and now the Decepticons can write it anywhere they want, right, they can write XC why the BA whatever order. It doesn't really matter. Okay. Because now I made a guarantee that

181
00:25:20.550 --> 00:25:31.890
William Cheng: You know that this transaction has been committed that I went to the next transaction, the transaction says E and F and G before I write g onto the onto the journal. Now I have a crash.

182
00:25:33.840 --> 00:25:41.340
William Cheng: Okay. So in this case, you know, XYZ ABC D which this block has gone to the actual bosses that I have no idea.

183
00:25:42.570 --> 00:25:47.940
William Cheng: Okay, so, so, so in this case I get a crash. Right. I mean, I release them to dissipate as it is somebody that's going to write them one at a time.

184
00:25:48.150 --> 00:25:55.470
William Cheng: I have no idea. Maybe none of that has gone to the actual versus then maybe all that has gone through the process them. So now when I reboot the foster sound. What, what, what should I do

185
00:25:56.220 --> 00:26:00.570
William Cheng: Okay, so one things. So there's not enough along to the desk. There is no coming record over here.

186
00:26:01.050 --> 00:26:10.320
William Cheng: Okay. So, therefore, what happened what I can do over here is that when I reboot the system. The first thing I need to do is that before I used to foster them because the false. So, now can be in the inconsistent state.

187
00:26:11.580 --> 00:26:16.530
William Cheng: Okay, how do you know the file system can be an inconsistency because I will find out that my journal is not empty.

188
00:26:17.490 --> 00:26:28.410
William Cheng: Okay, so every time when you pull the file system. The first thing you need to do is to check whether the journal is empty or not. If the journal is not empty, that means I'm in. I'm in the middle of something right over here, you can see that I'm in the middle of E and F.

189
00:26:29.220 --> 00:26:37.890
William Cheng: Okay, so therefore, in this case, there are transactions that the journal. So what I need to do is I need to actually release all these transaction into this up a task and tell it to update the desk.

190
00:26:38.700 --> 00:26:42.720
William Cheng: Okay, so don't forget what I will do is that I will go to the journal over here find

191
00:26:43.020 --> 00:26:50.670
William Cheng: The committee record, right. So the first company record over here. I'll see is the one with x y AMP z. So once I found this company record over here. It says, Oh, there's a committed transaction.

192
00:26:50.910 --> 00:27:02.100
William Cheng: I don't know if x, y, and z has been written out to the desk, but these are after images. So therefore, again I will send them to this update has to say write it down to this anytime you want. Right. And then I will continue.

193
00:27:02.610 --> 00:27:12.510
William Cheng: Looking at the journal and find a next come in record, say, Oh, I found a company record over here. So, A, B, C, D, they have to go to the new state, right. So again, I'm going to release this one through this update task and this

194
00:27:13.050 --> 00:27:16.290
William Cheng: Task and right there any way they want. What if I get another crash right now.

195
00:27:17.400 --> 00:27:20.670
William Cheng: What there's no difference between this question. The next crash and next time when I

196
00:27:20.940 --> 00:27:26.820
William Cheng: You know, restart. Again, I will see that the journal. Again, it's not empty. So therefore, there might be transactions and now

197
00:27:27.030 --> 00:27:38.640
William Cheng: I'm going to go there again look at the trend coming record over here says, Oh, there's a committed record x y AMP z i will release it with this update tasks and then I will go ABC and D. And then again, I see the company record I release it.

198
00:27:38.850 --> 00:27:42.600
William Cheng: And then when I tried to look for the next coming record, I get a crash again.

199
00:27:43.230 --> 00:27:54.660
William Cheng: I mean, you're not supposed to get that many crashes right but but again and I said that the power is unstable you keep getting crashes over here. So I'm gonna do this over and over and over again. Eventually ABC do X, Y AMP Z. They all have to go to the actual file system.

200
00:27:56.010 --> 00:28:03.720
William Cheng: Right. I mean, do you agree or sooner so so again I find a, b, c, d. And then I release it at this company to us. And then I won't be able to find another committee wrecker

201
00:28:04.350 --> 00:28:14.400
William Cheng: Okay, so therefore, now I have to wait until all these disbarred to go on to the actual process right that this of the task they might write this in any order. But eventually, that this update as well. I've nothing, nothing else to do.

202
00:28:15.270 --> 00:28:23.640
William Cheng: Okay, so when did this update has has nothing else to do, it will tell me that are say hey you know all these data has gone on to the desk and now your file system is in a consistent state.

203
00:28:24.360 --> 00:28:38.190
William Cheng: Consistency again. Why do you know that the you know the facilities and consistency, because I just finished running two transactions over here. I've taken the file system to a new state, right. And then if you're running transaction. I guarantee to be in a consistent state.

204
00:28:39.690 --> 00:28:47.580
William Cheng: Okay, so. So again what all these this block has written out to the desert. I know the file system in a consistent day. Now, what can I do, I can delete the journal

205
00:28:48.300 --> 00:28:57.060
William Cheng: Guys, I will go to the general over here. I'll delete everything so E and F and G over here, you know, again, it's written in the transaction and this guy's E F and G THAT WERE NEVER DONE RIGHT. SO THEY'RE IN

206
00:28:57.450 --> 00:29:05.250
William Cheng: The faster than is going to contain the previous day. And now when I leave the journal over here and now the general become empty now can continue to do the file system.

207
00:29:06.570 --> 00:29:08.400
William Cheng: Okay, so, so every time when you would often

208
00:29:08.880 --> 00:29:18.750
William Cheng: Every day, every time when you start restart operating system. The first thing you have to do is to check whether the journals empty or not. If the journal is not empty, then you have to perform this operation is known as recovery.

209
00:29:19.380 --> 00:29:30.510
William Cheng: Right, because you know that you know there's a crash in the middle somewhere. So you need to perform recovery, once you fix up the file, file system. The file system will be inconsistent state. Now you can delete a journal and you can actually continue with booting of our system.

210
00:29:32.130 --> 00:29:37.920
William Cheng: Or so hopefully you're convinced that, you know, if you do it this way, you know. The Fosters and will always be in a consistent state. Yeah.

211
00:29:40.470 --> 00:29:40.890
William Cheng: All right.

212
00:29:41.400 --> 00:29:53.460
William Cheng: So, so I guess the picture over the little show you write you write x, y, and z and also X y&z doesn't have to be any water right because it's all or nothing. So, you can write any water and then you release it to the, you know, to the dissipate HAS OVER HERE, JOHN us x y AMP z.

213
00:29:53.760 --> 00:30:01.020
William Cheng: So again, these are 30 blocks that this update tasks will start writing them, although this in any order that it wants. After the coming rock has been reaching out to the desk.

214
00:30:01.680 --> 00:30:14.340
William Cheng: Okay. So, therefore, what you would do is that you know you're, you know, he saw your process that if you're running generally periodically. What you need to do is that you need to look at all the transaction and perform this operation, you know, over and over again until you you know

215
00:30:16.950 --> 00:30:21.600
William Cheng: What you will do that you need to wait for this update as to finish. If it turns out you don't get a crash.

216
00:30:22.530 --> 00:30:29.610
William Cheng: That. So let's say that you write x y&z A, B, C, D, E, F, G, all these kind of stuff and then that this of the tasks. I know I'm all done.

217
00:30:30.600 --> 00:30:40.950
William Cheng: Okay. Also, at this point, you can actually lead in her journal, right, because you now you know that the file system is inconsistent day. I have no need for my journal anymore. So, therefore there. You can also add the journal

218
00:30:41.370 --> 00:30:49.680
William Cheng: Okay so journalists this I'll get the property of the journal is that it's a pen only never modify and you have another operation is to clear out the entire journal

219
00:30:50.010 --> 00:30:55.680
William Cheng: Okay, when can you close to the point when can you cloud entire journal when the this update task has nothing to do

220
00:30:56.040 --> 00:31:05.760
William Cheng: Right, because all your data blocks over here. They're always nothing. This is nothing to do. Now you know that that the file system is the consistency you can er free free to delete the jail. Okay.

221
00:31:07.890 --> 00:31:20.580
William Cheng: All right, so, so again, there are more texts will be here. You know, so you know what if you're using a review journal when you're doing the recovery, you look for you go to the journal, you look for all these, you know,

222
00:31:20.940 --> 00:31:29.460
William Cheng: Commit wrecker and then when you find a company record you can write all the after images, you know, after the decision any water that you want, man. Eventually when it's done.

223
00:31:30.030 --> 00:31:39.870
William Cheng: In that case, you're going to end up, you know, with a consistent office system. Okay, so, so, so what if the you know the the what's inside of transaction is increment.

224
00:31:40.320 --> 00:31:50.760
William Cheng: You know account number one by $100 go account number two by $100. Okay. So in this case, if you'd keep performed this redo operation over and over and over again, you're going to be very, very rich

225
00:31:51.240 --> 00:31:59.580
William Cheng: You know, because you keep incremental, you know, to come by, by $100 so therefore operations that incremental detriment cannot be part of a transaction.

226
00:32:01.320 --> 00:32:10.170
William Cheng: Okay, so, so you got to be very careful of what you want to put inside of transaction certain kind of operation. If you do if you put in there. Then you'll fastest and we're going to inconsistent state.

227
00:32:10.980 --> 00:32:18.360
William Cheng: Okay. So, therefore, what you're required to do, is that the only operation that you a lot of points that a transaction or no as item potent operation.

228
00:32:18.750 --> 00:32:28.080
William Cheng: Okay, it's kind of a fancy word. What it means is that for item potent operation doing it once. It's the same thing as doing it many, many times or doing it many, many times, the same thing as doing it once.

229
00:32:28.440 --> 00:32:38.700
William Cheng: Okay, so for example x plus plus there is this item potent operation. Well, no, because doing wise is not the same as doing it 100 times, right. What about x equals five.

230
00:32:39.360 --> 00:32:42.660
William Cheng: X equal to five, doing the ones the same thing as doing it 100 times

231
00:32:42.990 --> 00:32:45.300
William Cheng: Okay, so therefore this is it important operation.

232
00:32:45.540 --> 00:32:59.100
William Cheng: This is not a default operation. So that's why we try this, you know, a store data into the the journal we store the after image, right, because the after image you can write it down to this as many times as you want. It's the same thing as right into the desk exactly once.

233
00:32:59.940 --> 00:33:08.040
William Cheng: Okay. So, therefore, you know, using the redo, you know, redo generally works very well because all the operations over here or item potent operations.

234
00:33:08.940 --> 00:33:20.160
William Cheng: That alright so after you put on the recovery operation, instead of the file system is what it was at the end of the last committed transaction. So by definition or by the asset property is going to be in a consistent state.

235
00:33:21.210 --> 00:33:21.480
William Cheng: Okay.

236
00:33:24.210 --> 00:33:32.370
William Cheng: All right, so, so, so in this case, you know, when you get a system crash. The only time that matter when you crash is

237
00:33:32.670 --> 00:33:39.300
William Cheng: It's whether you crash before the commitment record has gone on to the journal or after the coming record gone through the journal right

238
00:33:39.540 --> 00:33:51.960
William Cheng: There. So let's say that if you crash before to come in and record in through journal. So in this case, there's no commitment record over here. So, since there's no come in the record, you know that X, Y, and Z inside the actual file system, there are still in the state.

239
00:33:53.160 --> 00:34:03.480
William Cheng: Right. There's no way for x, y, and z over here inside of our inside actual process, then there's no way that they are different from the previous day because I haven't released X, Y, and Z to this update task.

240
00:34:04.380 --> 00:34:07.230
William Cheng: Okay. So, therefore, you know, there's no way for X y&z to change already

241
00:34:08.100 --> 00:34:17.070
William Cheng: Right. Because, because, again, the only way to modify the file system is using transactions, while. So in this case, you know, since the commit record is not, it's not inside journal

242
00:34:17.280 --> 00:34:25.470
William Cheng: I know that x y AMP z will be in the previous day. So again, by the definition of the transaction, the previous day is is actually a consistency. Right.

243
00:34:26.430 --> 00:34:33.600
William Cheng: All right. What if you crash anytime after it works. Right. So as soon as you verify that are coming record has gone into the journal. Now you can crush it anytime

244
00:34:33.780 --> 00:34:37.950
William Cheng: And I'm guaranteed by using the algorithm that I mentioned when whenever you reboot the system.

245
00:34:38.130 --> 00:34:47.430
William Cheng: I'm going to go into the recovery state and instead of recovery stay. I'm going to write these data blogs on to this as many times as I need until all of them gone through this. So this way, they will go into the new state.

246
00:34:48.120 --> 00:34:59.130
William Cheng: Okay, so therefore, whether you crash before the commander occur this written out to the journal after the commitment of records would not the journal your guarantee that your file system will always be in a consistency.

247
00:35:00.210 --> 00:35:00.480
Now,

248
00:35:06.570 --> 00:35:11.640
William Cheng: Alright. So again, if you look at your desk right here. This has the you know the this is the

249
00:35:12.060 --> 00:35:14.280
William Cheng: This is the blue block and this is a super block.

250
00:35:14.460 --> 00:35:24.240
William Cheng: And he's had a superblock now you need to keep track of where your journal is where your actual file system is where the swap is so all that information is keep track of the super blah and inside the journal. We hear you know the journal is

251
00:35:24.750 --> 00:35:32.160
William Cheng: You know, needs to be pretty bad if you want to, you know, be able to store all that before and now all the after images inside the journal

252
00:35:32.910 --> 00:35:39.660
William Cheng: So, and the actual file systems on on the right over here. Right, so get x y AMP z over here will be the new image of the this block.

253
00:35:39.960 --> 00:35:51.210
William Cheng: Right, you read them out to the journal. First you got to come in record. Then you release all these data block to be written now into the actual file system. Anytime you anytime that this update tasks you know will wants to do it. Yeah.

254
00:35:53.490 --> 00:36:03.210
William Cheng: All right, so, um, so, as it turns out, generally. So what if you download a, you know, two gigabyte power 1.5 gigabyte file, file I want to 16.04

255
00:36:03.600 --> 00:36:13.440
William Cheng: Okay, if you're downloading to assist them and now you need to create a transaction to store that entire data, you know, inside the inside. Inside the journal. So now the journal has been 1.5 gigabytes in size.

256
00:36:14.190 --> 00:36:20.610
William Cheng: Okay, so this is going to be really, really big. Right. Okay. So, so, you know, again, generally they're true way to go. What is the journal everything

257
00:36:20.970 --> 00:36:30.720
William Cheng: Okay. Every little modification. I need to go into the file system, you need to journal that and then if you'd rather coming record you can you can actually release them to be written after this, in this case, your journal has to be really, really big.

258
00:36:31.830 --> 00:36:36.300
William Cheng: Okay, so some system says, you know, let's not do that. We can also journal meta data only

259
00:36:36.690 --> 00:36:46.620
William Cheng: Okay, we are actually so so in this case we're going to journal all the directory files. Okay, and all the I knows and all the other stuff. The only thing that we don't journal is the regular files content.

260
00:36:47.670 --> 00:36:51.840
William Cheng: Okay, so, so all the indirect, you know, is remembering system of US Assam there's these

261
00:36:52.950 --> 00:36:59.790
William Cheng: This map that this map is 13 point or something but direct pointer is on the main drag pointer. So all these pointer. They're all metadata.

262
00:37:00.000 --> 00:37:07.920
William Cheng: Right, which are picked up before I all those things are meta data. So we're going to journal all the metadata. So and so, but we're not going to general the data because the data is really big.

263
00:37:08.400 --> 00:37:15.840
William Cheng: Right. If you download a 1.5 gigabyte file, the data is going to be 1.5 gigabyte. What about the. What about the metadata. Well, the meta data tends to be quite small.

264
00:37:16.620 --> 00:37:24.000
William Cheng: Okay, so therefore, in this case, you know, our file system is not going to have too much overhead and also if you want to copy all the data into the journal that also going to slow things down.

265
00:37:24.420 --> 00:37:32.400
William Cheng: Okay, so, so, so there are some system. They actually provided an option whether you want a journal everything. So in that case, you need to reserve a really, really large space for the journal

266
00:37:32.820 --> 00:37:36.540
William Cheng: Or you can do my journal meta data only. So in this case, your journal is going to be small.

267
00:37:36.930 --> 00:37:48.210
William Cheng: Okay, so what about the actual data, the actual data is not journal. So again, in this case, when you try to demo the 1.5 gigabyte file. If you get a crash in the middle. The father you download that will be, you know, can

268
00:37:48.750 --> 00:37:57.510
William Cheng: Can be bad, right, because, because the you know the the pointer, the metadata will be journal, so therefore you will find all these metadata when you try to access the file the file will be totally corrupted

269
00:37:58.680 --> 00:38:05.100
William Cheng: Okay, so he's not okay. That's okay if you are, if you know in that you're you're generally only metadata. Well, in that case, that will be allowed.

270
00:38:06.480 --> 00:38:12.300
William Cheng: Okay, so. So again, just like what I said before, if you're editing a Word file. The last modification that you may

271
00:38:12.810 --> 00:38:21.150
William Cheng: The you may, and then you, you click on Save at the time at the exactly the same time that you lose power. It is possible that you'll fall will be corrupted

272
00:38:21.570 --> 00:38:31.620
William Cheng: Guys, so therefore, again, if you turn on metadata only it is possible. You're gonna lose that file. So therefore, even though if you're usually generally, it's still very, very important that you backup your that the backup all your work.

273
00:38:32.670 --> 00:38:44.430
William Cheng: Or I mean, some people say, oh, usually generally these crash proof. So therefore, I don't have to, you know, I don't have to back up. You still have to do backup. They're also another reason why you have to do backup. What if you lose the entire disk.

274
00:38:45.690 --> 00:38:55.440
William Cheng: Okay, so for some reason. You know, I have some hardware issues, you know, the machine is getting old and now when I turn off my power, you know, somehow I scratched it is the best scratcher this the entire district on useless.

275
00:38:55.770 --> 00:39:02.850
William Cheng: Because they'll get if you don't have backup is going to be in trouble. Right. I think some people also income to the same problem with a virtual machine, the virtual machine. Sometimes when you

276
00:39:03.120 --> 00:39:08.490
William Cheng: You know, some people when they tried to, you know, sort of turn off their machine or when they lose power on their machine. They don't really shut

277
00:39:08.730 --> 00:39:12.510
William Cheng: You know, sort of shut down their virtual machine properly the virtual machine is really, really big.

278
00:39:12.900 --> 00:39:19.080
William Cheng: Right, so if you lose power at the time when the virtual machine tried to shut down. It is possible, your virtual machine is going to go into inconsistent state.

279
00:39:19.650 --> 00:39:23.460
William Cheng: Okay, so therefore again next time we tried to put the virtual machine, the virtual machine will refuse to boot.

280
00:39:23.730 --> 00:39:35.550
William Cheng: All the files that you store in the virtual machine will be lost. So again, it's still very, very important. The backup data because we don't have a system that will never lose data, right, the only kind of system that I know that doesn't lose data is the banking system.

281
00:39:35.940 --> 00:39:48.630
William Cheng: Right. So again, they spend a lot of money to make sure they could build a system that no matter when you crash, you can always go back to the previous day. OK. So again, you know, regular, you know, the general purpose, the organism. I don't generally do that. Yeah.

282
00:39:50.370 --> 00:39:53.580
William Cheng: All right, so, so, you know, sort of defense system over here. Again, they have a

283
00:39:55.620 --> 00:40:06.690
William Cheng: Sort of a generally option. So again, over here reminders. They, in general, it's extremely costly. If you want to make sure that data is never lost. Okay, so if you lose power at the same time you try to save a document.

284
00:40:07.200 --> 00:40:10.380
William Cheng: Did you click on Save right before you right after the power loss.

285
00:40:11.010 --> 00:40:21.210
William Cheng: Okay, so he so there's no way for you to know right if I click on safe. Did I do this right before the power loss or right after the power loss. Right. So if you want us to sort of see Microsoft say hey you know when I tried to save your file.

286
00:40:21.630 --> 00:40:27.270
William Cheng: You enter you current my file Microsoft it true to me that you actually, you click on Save you know after the power loss.

287
00:40:27.780 --> 00:40:31.860
William Cheng: Once you actually click on Save before the power loss. Well, there's no way for you to actually prove that

288
00:40:32.460 --> 00:40:40.950
William Cheng: Okay, so therefore, in the end, you know, it is always a possibility, you know that your modification didn't go into the file system that unless you're willing to pay pay a lot of money. Right.

289
00:40:43.260 --> 00:40:48.300
William Cheng: All right, so, you know, the Linux has, you know, different pastas them right and

290
00:40:49.200 --> 00:41:01.830
William Cheng: I don't know if you have heard of these are called EFT to EFT 364 X 32 is the FS calm rising remember effort versus the fast processor file, file systems improvement on or, you know, over system, our bosses them.

291
00:41:02.310 --> 00:41:08.220
William Cheng: To use cylinder group we use all these kinds of tricks to try to use you know 100% of this capacity, but can only achieve

292
00:41:08.640 --> 00:41:13.020
William Cheng: You know 50% of this right capacity right that's that's the, that's the bathhouses, then

293
00:41:13.710 --> 00:41:22.170
William Cheng: Also, we're going to talk about, you know, other kind of improvement that was done the fastest the fastest ism is a very important file system in the Unix operating system guys a Linux.

294
00:41:22.650 --> 00:41:27.720
William Cheng: They re implement the fast else's then the column, E, F, G two. Okay, so yes teacher was the fastest

295
00:41:28.350 --> 00:41:35.580
William Cheng: The fastest losses and will also mentioned that in 1990s, they start using software update for crash resiliency.

296
00:41:35.970 --> 00:41:44.040
William Cheng: Guy so yesterday to will also use self update. So if you are so we need to configure a Linux is them. I guess if you try to install Linux on

297
00:41:44.580 --> 00:41:53.220
William Cheng: On the actual hardware, it will actually ask you what kind of forces them. You want to put on it. If you choose the XT to you will notice that you have a slash Lost and Found a directory in the

298
00:41:53.550 --> 00:42:07.680
William Cheng: In the route last last plus found all right in the road because they're using software update. Okay. The other buses. I'm over here is either 6364 X 33 is yes T to with journaling.

299
00:42:08.430 --> 00:42:18.450
William Cheng: Okay, so therefore UFC three is is quite resilient and you will also see that yes D3. There is no such directory in the route, right, because they're not using solve update

300
00:42:19.020 --> 00:42:29.310
William Cheng: That ESC for is an improvement over UFC three because they add additional features into UFC three now. So, therefore, if you're using Linux, you should always use EFT three or years before I mean

301
00:42:29.850 --> 00:42:44.790
William Cheng: Before is pretty stable these days. So you should be using EFT for now. So, so you actually three their support both food generally a mentor. They'd only generally it's actually an option that you can select when you try to what we install Linux. Yeah.

302
00:42:46.290 --> 00:42:56.520
William Cheng: So there are some additional details. So instead of running transaction. They call him sub transactions. So again, the basic idea is still the same. So, what it will do is that, you know, once in a while, you know, you

303
00:42:58.320 --> 00:43:05.790
William Cheng: Inside the Colonel, they have sort of a task, called the checkpoint tasks. So unfortunately they resist term, right, so remember in the journaling classes and there's something called check

304
00:43:06.990 --> 00:43:16.740
William Cheng: Some of your checkpoint file. Okay, so check on file is a data structure that you use to to keep track of all the bases implemented, I know map again. Then they use a double buffering technique.

305
00:43:17.220 --> 00:43:19.950
William Cheng: To make sure that if you crash, everything will be okay now.

306
00:43:20.400 --> 00:43:32.550
William Cheng: So in Linux. They call this operation called check pointing. So the basic idea here is to find all the transaction right of them only to the journal verify that they're all go on to the desk right so in that in the end over here. All the confirm

307
00:43:33.060 --> 00:43:44.400
William Cheng: All the committed transaction and they have been written out to the desk and now you can clear the journal. Okay, so for Linux. They called us pretty good operation. They call the check check pointing at the end of chapter 19 you can actually clear the journal

308
00:43:45.660 --> 00:43:51.840
William Cheng: Alright, so again, they use similar to the terminology. They use journaling. That sounds like a lot of houses, then began there and they're not very different

309
00:43:52.500 --> 00:43:55.230
William Cheng: Because, you know, sort of computer science people keep reusing the same terms.

310
00:43:55.980 --> 00:43:59.430
William Cheng: As well. Yeah. Some people will confuse journaling with the log structured analysis then

311
00:43:59.760 --> 00:44:10.290
William Cheng: The last shot you ever since then. The purpose of the log structure forces them to demonstrate that you can actually achieve 100% or Neil 100% of the right throughput of the desk. Okay.

312
00:44:11.220 --> 00:44:19.620
William Cheng: You know, for generally the purpose of generally is simply to provide crash resiliency. They're not trying to prove that you can write a new 100% of the district capacity.

313
00:44:20.010 --> 00:44:28.590
William Cheng: Good for the law structure of us and they use the checkpoint file to perform coarse grain recovery. So why does it say this is coarse grain road recovery.

314
00:44:28.860 --> 00:44:36.540
William Cheng: Because, you know, because what it will do is that you need to wait for a lot of data to to be written out to the log before they perform one long right to the to a lot

315
00:44:36.990 --> 00:44:44.970
William Cheng: Okay. So in this case, you might actually have to wait for 10 seconds even 30 seconds before you write data to the law. So we ran a little long, you do a very, very long right

316
00:44:45.720 --> 00:44:53.730
William Cheng: Okay, so therefore, in this case, when you recover if you get a crash in the middle, you're going to go back to the previous day, the previous day might be 30 seconds later, instead of five or 10 seconds later.

317
00:44:54.060 --> 00:45:04.350
William Cheng: Okay, so therefore the recovery over here is coarse grain because it allowed data can reach out to this guy if he was in generally generally also the log structured process that is a file system.

318
00:45:04.710 --> 00:45:07.110
William Cheng: It cannot be added to any existing file system.

319
00:45:07.560 --> 00:45:20.520
William Cheng: Okay. On the other hand, Journaling is a technique, it can be added to any existing process. Then, as it turns out, all the modern audiences and they use, you know, the gentleman technique that they use check pointing basically to perform right back.

320
00:45:21.330 --> 00:45:29.520
William Cheng: Okay, so when you try to read all the data, you know, under the desk right that we will call you right back. Right. So the two technique ones go right through the other ones go right back when you perform right back well

321
00:45:29.970 --> 00:45:39.990
William Cheng: For Linux is during the checkpoint operation. That's where you find all the transactions right them all to the journal commit all these times actually wait for all the data gone through this and then you clear the journal

322
00:45:40.830 --> 00:45:48.570
William Cheng: OK, so now case they call the checkpoint. Right. So again, the other main reasoning to do this is that now the fastest them is a consistent day. And now you can see the journal

323
00:45:49.020 --> 00:45:55.620
William Cheng: Okay, so this way, the journal can be reused over and over again. So the journal is not the same as a lot shorter file system. Yeah.

324
00:45:58.560 --> 00:46:04.290
William Cheng: Alright, the next thing that we want to briefly talk about his shadow page. And that's another way to rent rent. Rent says that transaction.

325
00:46:04.890 --> 00:46:16.230
William Cheng: It's, it's sort of a different technique overhears it's it's a refreshingly simple it's based on the idea of copy on right so what is copyright right so a copyright is done is done on memory.

326
00:46:16.800 --> 00:46:23.460
William Cheng: Okay, so we're going to use the same technique inside of our system. Right. So again, this case, you know, whenever you try to make changes to it this block.

327
00:46:23.790 --> 00:46:34.020
William Cheng: Okay, when he tried to make it make changes to the display for the first time you make a copy you right into the copy right so this way, the original data is considered read only. They, they actually they would never get changed.

328
00:46:34.680 --> 00:46:42.060
William Cheng: Okay, so, so we're going to see how you can actually do this on the file system. Yeah, there's some people use yellow pages. One is

329
00:46:42.720 --> 00:46:48.780
William Cheng: waffle houses, then this one was done by the network appliance and then we'll appliance. They are very famous for, you know,

330
00:46:49.350 --> 00:46:57.600
William Cheng: Doing network attached storage. Basically, they are the one that come up with a design, you can actually plug it into your network and now your entire house can actually use that storage device.

331
00:46:57.990 --> 00:47:09.660
William Cheng: That actually they can plug inside of network. Yeah, another one. It's called CFS. That one was, you know, what was done by Sun Microsystems Sun Microsystems later on got bought out

332
00:47:10.800 --> 00:47:14.760
William Cheng: You know by Compaq and they got merged with HP, you know, and the

333
00:47:17.730 --> 00:47:28.890
William Cheng: Way, they're the one that you own by by Oracle now. Right. So, so I guess you know SUN. Sun Microsystems is the one that they invented Virtual Box as a Virtual Box and as part of Oracle because I'm ecosystem was

334
00:47:29.340 --> 00:47:34.890
William Cheng: I guess I was thinking about different company. So Mike was talking about digital equipment Corp as well. So, so some ecosystem.

335
00:47:36.030 --> 00:47:43.320
William Cheng: And now it's owned by Oracle. So, therefore, you know, the 50 other devices then be able to work. Oh yeah, I remember that at some point.

336
00:47:44.100 --> 00:47:52.530
William Cheng: When Mac OS X, when they first released, they were thinking about using DFS as their file system, but in the end they change their mind they decide to use a

337
00:47:53.280 --> 00:48:01.170
William Cheng: Different file system. And they also use generally to protect the other file system. Okay. They could have choose chose CFS. But it didn't go that way.

338
00:48:01.980 --> 00:48:07.560
William Cheng: Alright so let me shut us to the show you. I'll demonstrate how to actually use the idea of copy on write. He's not a file system.

339
00:48:08.190 --> 00:48:16.650
William Cheng: If you look at all the this blog inside your file system hierarchy, right. So to get inside your actual file system, there is a file system hierarchy, right, and then they're also the free blocks.

340
00:48:16.920 --> 00:48:28.650
William Cheng: Okay, so what is the most of them are hierarchy, right. So the file system hierarchy, you can sort of think about is that if you start from the route, I note. Okay, if you start from the road. I know you will be able to find all the this blog inside of our system hierarchy.

341
00:48:29.880 --> 00:48:33.960
William Cheng: Right, because you know your route, blah. You're rude. I noticed, I know you open now you're gonna find all these

342
00:48:34.230 --> 00:48:41.460
William Cheng: All these other I notes. Some of them are directory. Some of them are file. So again, it's a recursive data structure, you can go to all these directory. You can go to files.

343
00:48:41.640 --> 00:48:51.330
William Cheng: You open up the I know you're going to find the you know the this map, you're going to find all these data block. So if you start on the road, I note over here you will be able to find the entire file system hierarchy with every discipline.

344
00:48:52.680 --> 00:48:58.380
William Cheng: Alright, so, so from this point on, we're going to perform. I'm going to use the ideal copy on write well you need to modify this block.

345
00:48:59.160 --> 00:49:06.750
William Cheng: Okay, so all the yellow ones over here, we're all going to consider them read only. Okay, so this way when we try to write it for the first time. Right, we're going to make a copy. I'm gonna

346
00:49:07.170 --> 00:49:11.130
William Cheng: We'll get right into the copy. Right. So for example, let's say we want to modify this this block.

347
00:49:11.700 --> 00:49:17.340
William Cheng: Okay, so in order for you to modify this. This is what do you have to do, right, you have to allocate a new this blog over here, make a copy of it.

348
00:49:17.760 --> 00:49:22.770
William Cheng: Right. And now you need to make changes inside this as far. Okay, so when you're done, he will look like this.

349
00:49:23.040 --> 00:49:33.150
William Cheng: Okay. So the blue one is the one that's me modify the yellow one is original data. Okay. But if the if the first doesn't look like this, you will never be able to find this blue block, right, because if you start from the root.

350
00:49:33.360 --> 00:49:41.250
William Cheng: This blue block is not inside the file system hierarchy. So what do you have to do what this point or is pointing in the wrong place, right. This one is funny here, it should be plenty here.

351
00:49:42.300 --> 00:49:49.080
William Cheng: Well, so therefore, in order for us to modify this this pointer inside this this blah, we need to apply copy on write principled again.

352
00:49:49.290 --> 00:49:53.790
William Cheng: We need to allocate a new this block and make a copy of this one will you make a copy, then

353
00:49:54.060 --> 00:50:01.290
William Cheng: The point to point to the same place. So now inside this new this blog, there's going to be two pointer at one point of this this blog and the other one pointed this this block.

354
00:50:01.500 --> 00:50:05.850
William Cheng: And now we're going to modify the copy over here we're going to change this pointer to point to the new data right here.

355
00:50:06.270 --> 00:50:14.040
William Cheng: Okay, so therefore, when we're done. It will look like this right here is a new this block the this point, just go punch the same place as the other point. Yeah, this one will point to the new the spot.

356
00:50:14.970 --> 00:50:20.970
William Cheng: Okay, but when we do this after we finished doing this again, we're not going to find either of these this block inside of our system hierarchy.

357
00:50:21.150 --> 00:50:29.610
William Cheng: Because this point or is pointing in the wrong place, right. So, therefore, again, we don't apply copy on write allocate a new this block copy it over and now this point it will point right here.

358
00:50:29.820 --> 00:50:33.600
William Cheng: We need to change this point to point right here and now it will look like this. There

359
00:50:33.780 --> 00:50:41.160
William Cheng: But this point or is pointing in the wrong place. We're going to allocate a mute this block over here, copy over and now the left point, at what point here, the middle point oh point here.

360
00:50:41.370 --> 00:50:48.960
William Cheng: The right point oh point here, we modify the copy. So they will apply right here and now it will look like this. Right. So again, all, none of the book can be found.

361
00:50:49.170 --> 00:50:56.310
William Cheng: So finally, we need to make a copy of the route, I note over here so that the first point to a point here. And the second part of a point here and now we Paul

362
00:50:56.940 --> 00:51:03.030
William Cheng: The second point over here upon to the new new this block and now it will look like that. And now you have a system crash, what's going to happen.

363
00:51:03.720 --> 00:51:08.100
William Cheng: If we have a system crash will go back to the previous version of the fastest then because this will be the route.

364
00:51:08.400 --> 00:51:13.650
William Cheng: All we can see our yellow blocks all the other blue blocks. They are innocuous in consistencies

365
00:51:14.010 --> 00:51:23.550
William Cheng: Okay, so there won't be able to find that. So there's one more thing that we need to do in order for us to move to the new fastest and we need to go to the super blog and without a superblock to say this is the route. I know.

366
00:51:24.450 --> 00:51:31.020
William Cheng: Okay, so in that case we need to write the superblock back to the days when we write a supervisor back onto the this that's like a commitment.

367
00:51:31.890 --> 00:51:38.160
William Cheng: Okay, so at the time we write this word log onto the days, we need to verify that this has been written out to this when we do that, that's our commitment time

368
00:51:38.340 --> 00:51:42.930
William Cheng: Because if we have a crash before the commitment time we go back to the previous state of the foster Sam.

369
00:51:43.170 --> 00:51:48.840
William Cheng: If we have a crash after the commitment time. In that case, when we reboot the system. So if you've got a crush after we made the commitment.

370
00:51:49.020 --> 00:51:55.290
William Cheng: Next time, when we reboot. We're going to go to the super blog. Find out where the roof SMS and now we will find the new state and the Father, Son,

371
00:51:55.500 --> 00:51:58.770
William Cheng: And by definition over here at the new stay over here will be a consistent state.

372
00:51:59.010 --> 00:52:07.620
William Cheng: Right. Is there any way we can actually go back to the old state. Well, then there's no way to do it. We need to free up all these old blocks over here. So again, we're going to have, you know, something very similar to solve update

373
00:52:07.830 --> 00:52:14.490
William Cheng: We're going to have a disguise scavenger thread inside the colonel though actually find all these blocks and and free them up and return back to the free list.

374
00:52:16.110 --> 00:52:26.640
William Cheng: Alright, so, so this is called Coco shadow paging. So these are called shadow pages. So again, we're gonna move. Basically, we use the copy on write idea on this block instead of memory. Okay.

375
00:52:28.170 --> 00:52:37.650
William Cheng: All right, so the relocation is written. It's like a commandment record. So therefore, again, this is this is this is some people consider this also running a transaction right again there's running transaction is very, very different.

376
00:52:37.890 --> 00:52:46.110
William Cheng: From the database kind of transaction, we have, you know, we actually have a we have a transaction which is like a data structure and you put a bunch of data into the transaction. Yeah.

377
00:52:48.150 --> 00:52:57.900
William Cheng: Alright, so this is a good time to break. So next time, I'm going to continue and talk about some other changes that will make inside the fast file system. Okay, so I will see you in a bit.