12:20 < joe9> BurnZeZ: do you recall how you found out about replica(8). It is cool. I cannot believe that it is not more popular 12:22 < joe9> anyone has the man books? 9FRONT PROGRAMMER'S MANUAL V2 and V1. I am about to order them, but, want to check if anyone has them and find them useful. 12:24 < mycroftiv> joe9: old plan 9 from bell labs used replica/pull as the mechanism to update the distribution 12:25 -!- Irssi: mycroftiv myrkul 12:25 < kvik> khm: ok, good to know. my domain and patience with gandi webshit is about to expire, so i'm looking around 12:25 < joe9> mycroftiv: do you use it? any practical experiences to share? 12:25 < joe9> kvik: namesilo? 12:25 < aiju> we specifically dropped replica because it sucked 12:25 < mycroftiv> joe9: i like burnzez rewrite of the support scripts to 'rep' 12:25 < aiju> i think it was very slow 12:25 < aiju> but i forgot what other objections there were 12:26 < aiju> obviously centralised 12:26 < mycroftiv> the value of replica(8) as a tool is independent of whether or not it was well used and administered by labs for distro updates 12:27 < mycroftiv> the replica(1) scripts and config are rather over-elaborate which is why i think it hasnt been used very much by people for personal file tree synchronization 12:27 < kvik> my main objection is it's grossly overcomplicated to setup for what you end up with 12:28 < mycroftiv> thats why i like burnzez rep stuff, but since it is undocumented and exists only in burnzez griddisk, it hasnt taken over the world yet 12:28 < joe9> I have data on a hosted vm, my local machine, backup of the local machine. I used fcp -x and disk/mkfs to synchronise them. but, disk/mkfs only uses modification time. Hence, I had to have a wrapper to ensure that I am not missing stuff. 12:28 < kvik> i should really implement proper sync for clone. but i've been saying that since the winter 12:28 < joe9> when reading about replica, it seems to do exactly that. 12:29 < joe9> but, I am not sure how it is regarded in practice. 12:29 < mycroftiv> yes syncing files between systems is exactly what replica is intended for 12:29 < kvik> yea, it also just compares timestamp iirc 12:29 < joe9> kvik: replica(8) says that it uses modtime and size. 12:30 < kvik> i guess there's just a lot more buttons to press as opposed to bare mkfs 12:30 < mycroftiv> its little used in practice because setting up the config files is one of those things where everyone reads the manapge and says 'uck, maybe ill deal with this some other time' 12:30 < joe9> mycroftiv: do you know where burnzez's scripts are? replica(1)? 12:30 < mycroftiv> joe9: they are in burnzez griddisk directory on the public grid 12:30 < mycroftiv> ive been trying to get him to document them for weeks but burnzez doesnt work to order 12:31 < joe9> that makes sense. I read the man page and realized that I need a print edition and understand them before I can do much. 12:32 < joe9> mycroftiv: do you know if burnzez has any plans to merge his scripts into 9front? or, does he want to leave them as they are. 12:40 < aiju> hm 12:40 < BurnZeZ> I do not because replica sucks 12:41 < aiju> yeah, i think the knowledge of knowledge ofknowledge of ... thing is spot on 12:44 < joe9> BurnZeZ: if you have a few free minutes, what do you not like about replica? speed? or hard to configure? 12:45 < joe9> I have a high latency link to my hosted vm. I use fcp -x to transfer files. 12:45 < BurnZeZ> While speed is not terrible, it's not great 12:45 < BurnZeZ> Configuration difficulty is one factor 12:45 < joe9> I want to replace it with replica(?) 12:45 < BurnZeZ> The other part is that replica seems to be designed with a parent-child dichotomy 12:45 < BurnZeZ> client-server 12:45 < BurnZeZ> But it doesn't make much sense to do so 12:46 < BurnZeZ> Really it should be bidirectional, and it has a bunch of nonsense to do just that 12:46 < joe9> my use case is that I always get stuff from the hosted vm. so, the client-server is fine here. 12:46 < joe9> and I am always sending stuff to the backup. 12:46 < BurnZeZ> If you do have synchronization needs, it works okay 12:47 < joe9> so, I have no need for bidirectional stuff. 12:47 < BurnZeZ> It breaks with append-only files 12:47 < joe9> oh, ok. I am not using append-only files. 12:47 < joe9> I had issues with append-only files too. 12:48 < joe9> even fcp or cp (I cannot recall) messed such files up. burnzez. do you know if the rep stuff is hosted somewhere else other than on griddisk? joe9 : burnzez. do you know if the rep stuff is hosted somewhere else other than on griddisk? BurnZeZ : Probably not I am not sure what the purpose of the log file is. in /n/griddisk/burnzez/rep/rep. But, it lists some files which are not in root/rc/bin/rep joe9 : I am not sure what the purpose of the log file is. in /n/griddisk/burnzez/rep/rep. But, it lists some files which are not in root/rc/bin/rep kvik : wewantrepdocs.com is free stillneedrepdocs : i had a few burnzez lines saved at one point i copy/pasted to explain things so i could probably grep the logs for them stillneedrepdocs : rep grep BurnZeZ : nuts BurnZeZ : I was going to write docs then got super demotivated because I realized replica’s problems are there any other problems? joe9 : are there any other problems? stillneedrepdocs : moody : BurnZeZ: do you have a quick reference for rep ? stillneedrepdocs : mycroftiv : BurnZeZ : mkdir /rc/bin/rep; cp /n/griddisk/burnzez/rep/rep/root/rc/bin/rep/* /rc/bin/rep/ stillneedrepdocs : mycroftiv : BurnZeZ : mkdir /usr/$user/lib/rep stillneedrepdocs : mycroftiv : BurnZeZ : rep/cinit /n/griddisk/burnzez/rep/libautism/ libautism stillneedrepdocs : mycroftiv : BurnZeZ : This line creates the directory /usr/$user/lib/rep/libautism stillneedrepdocs : mycroftiv : BurnZeZ : rep/pull libautism BurnZeZ : I don’t recall any at the moment mycroftiv : if i recall, you probably also want to rep/cinit for rep itself too 10:33 < joe9> BurnZeZ: I am familiarising myself with replica code. I have to profile it and figure out why it is slow. From a cursory reading, I would assume it is due to finddb(?). 10:33 < BurnZeZ> Hmm 10:34 < BurnZeZ> Replica shouldn't be slow 10:34 < joe9> on a local machine, I can see it going through each file. 10:34 < BurnZeZ> Ah 10:34 < joe9> replca/applylog is what is being run when I do replica/pull -nv 10:34 < BurnZeZ> You're using the replica scripts 10:34 < joe9> this is for the first time that replica runs. 10:34 < joe9> yes, I am using replica's scripts 10:35 < BurnZeZ> I wrote rep because I felt the original scripts were too complicated 10:35 < joe9> not yours yet. I am want to understand what was before before I can figure out your changes. 10:35 < joe9> oh, ok. 10:35 < BurnZeZ> It's probably worth learning replica(8) instead of the scripts 10:35 < BurnZeZ> Err 10:35 < joe9> rep is a replacement of replica(1)? 10:35 < BurnZeZ> Yeah, replica(8) 10:35 < BurnZeZ> joe9: Basically 10:36 < BurnZeZ> There is two things 10:36 < BurnZeZ> replica, and replica rc scripts 10:36 < joe9> but, it is not replica(1) that is the issue. I can see the replica/applylog going through each file. 10:36 < BurnZeZ> Yeah 10:36 < joe9> replica/applylog belongs to replica(8) 10:37 < BurnZeZ> Another bad thing 10:37 < joe9> to understand, rep is a wrapper around replica(8), correct? 10:37 < BurnZeZ> replica(8) is not fully documented 10:37 < BurnZeZ> There is undocumented parameters that are important 10:37 < joe9> the code looks in cinap's style. Did cinap write replica/applylog? 10:38 < BurnZeZ> It's bell labs stuff I think 10:38 < joe9> 10:37 < joe9> to understand, rep is a wrapper around replica(8), correct? 10:38 < joe9> would you mind answering it, please? 10:38 < BurnZeZ> joe9: One thing replica/applylog does that sucks 10:39 < BurnZeZ> Yes, rep is scripts that wraps replica(8) 10:39 < BurnZeZ> it spools files to /tmp/ first 10:39 < BurnZeZ> Which slows everything down 10:39 < Aram> aiju: ugh, I didn't know such things existed at all: https://mathoverflow.net/questions/43462/existence-of-a-smooth-function-with-nowhere-converging-taylor-series-at-every-po 10:40 < BurnZeZ> joe9: It may be worth updating replica(8) so that the manpage and code are synchronized 10:40 < joe9> it spools files to /tmp/ first - thanks, I see the big files in /tmp/. I cannot figure out what they are though. 10:40 < joe9> replica09550629 is the filename 10:40 < BurnZeZ> Yeah 10:40 < BurnZeZ> It's really stupid 10:40 < Aram> in fact 10:40 < Aram> https://en.wikipedia.org/wiki/Non-analytic_smooth_function 10:40 < BurnZeZ> I mean, I understand why it is done 10:41 < BurnZeZ> And it has a good reason 10:41 < BurnZeZ> But not good enough 10:41 < joe9> which file is it? Is it the original file from the server? 10:41 < BurnZeZ> Yes 10:41 < joe9> s/which/what/ 10:41 < BurnZeZ> It copies the server file to /tmp/ before replacing the client file with it 10:41 < joe9> oh, really. over a low latency link, that would be a killer 10:42 < joe9> even if the files are the same? 10:42 < BurnZeZ> No 10:42 < BurnZeZ> However 10:42 < BurnZeZ> It does not do delta synchronization 10:42 < joe9> I understand and do not want delta sync 10:42 < BurnZeZ> Just making sure that's clear 10:43 < BurnZeZ> If the server file is modified, the whole file must be copied 10:43 < BurnZeZ> And it copies to /tmp/ first 10:43 < BurnZeZ> I think the idea is so you can be sure you could actually GET the entire file 10:43 < BurnZeZ> Before you replace the client file with it 10:43 < BurnZeZ> Instead of start replacing the file and then some read error 10:44 < joe9> yes, that makes sense. 10:44 < joe9> In my case, this is the first time I am running replica and it should not be doing it when the files are the same. 10:44 < joe9> let me reset the client db and log and try. 10:44 < BurnZeZ> Hmm 10:44 < BurnZeZ> Yeah, it sounds like the database is not right 10:44 < BurnZeZ> That's part of why I wrote rep 10:44 < BurnZeZ> But after I wrote it, now I hate replica :) 10:45 < BurnZeZ> The best thing would be to use venti for all this 10:45 < BurnZeZ> Then we would even have delta synchronization 10:45 < BurnZeZ> For free 10:45 < BurnZeZ> venti concept, I mean 10:46 < BurnZeZ> Not the implementation that exists 10:46 < joe9> another thing that I cannot understand, When I start from scratch, no db or log file, I get this error 10:46 < joe9> replica/compactdb: opendb /dist/replica/client/ddfbkp.db: '/dist/replica/client/ddfbkp.db' does not exist 10:47 < joe9> this is probably replica(1) in play(?) 10:47 < BurnZeZ> touch the file first 10:47 < BurnZeZ> I don't think the scripts will create it 10:47 < BurnZeZ> I could be wrong 10:48 < joe9> yes, it works then. But, in that scenario, it skips the !havedb if statement in applylog 10:48 < joe9> I am wrong. 10:48 < BurnZeZ> Ah, right 10:48 < BurnZeZ> Oh 10:48 < joe9> sorry, my mistake, I think. 10:48 < BurnZeZ> Yeah, the scripts are confusing 10:48 < BurnZeZ> I recommend ignoring them and using replica(8) directly 10:49 < joe9> ok, thanks. will think this through. 10:49 < BurnZeZ> 9fs 9contrib && replica/updatedb -r /n/9contrib/contrib $home/lib/replica/contrib/sdb >> $home/lib/replica/contrib/log 10:49 < BurnZeZ> 9fs 9contrib && replica/applylog $home/lib/replica/contrib/cdb /tmp/contrib/ /n/9contrib/contrib < $home/lib/replica/contrib/log 10:49 < BurnZeZ> A couple old snippets 10:49 < yuuko> https://kurofuku.me/i/1559843282.png 10:49 < joe9> yes, please, these will help me a lot. 10:49 < BurnZeZ> This doesn't use replica(1) 10:49 < BurnZeZ> Even though it uses the same directory paths 10:50 < BurnZeZ> I think 10:51 < joe9> ok, thanks. 10:51 < BurnZeZ> I really dislike the replica scripts 10:51 < BurnZeZ> They are just way too complicated 10:52 < BurnZeZ> If it was up to me, I would get rid of them entirely 10:52 < joe9> ok, thanks. I will read up on your scripts now that I have a clue. 10:52 < BurnZeZ> All they do is confuse people who want to use replica 10:54 < BurnZeZ> ftpfs -a my@mail goes.gsfc.nasa.gov && replica/updatedb -r /n/ftp/pub/goescolor/goeseast/hurricane2/color_lrg -x older_images /tmp/replica/goes.gsfc.nasa.gov/sdb >> /tmp/replica/goes.gsfc.nasa.gov/log 10:54 < BurnZeZ> ftpfs -a my@mail goes.gsfc.nasa.gov && replica/applylog /tmp/replica/goes.gsfc.nasa.gov/cdb (/tmp/ftp/goes.gsfc.nasa.gov /n/ftp)^/pub/goescolor/goeseast/hurricane2/color_lrg/ < /tmp/replica/goes.gsfc.nasa.gov/log 10:54 < BurnZeZ> I doubt these will be valid anymore 10:54 < BurnZeZ> But it's another interesting usage 11:03 < joe9> to first seed the client db, is there a replica/scan on the client? 11:03 < joe9> s/is there a/is there a way to do/ 11:04 < BurnZeZ> You can run updatedb on the client root, and create the client database 11:04 < joe9> ok, that should help me. Thanks. 11:05 < BurnZeZ> But I'm not sure if this will confuse replica in some way 11:05 < BurnZeZ> Ideally it should work 11:05 < joe9> ok, thanks. I should focus on replica(8) then. Thanks. 11:33 < joe9> BurnZeZ: do you recall what this does? clientdb clientroot serverroot [ path ... ] 11:33 < joe9> it is documented twice in replica(8) too. 11:33 < joe9> I mean the command. 11:33 < BurnZeZ> It's the other commands being wrapped 11:34 < BurnZeZ> man -P 8 replica 12:33 < mischief> joe9: have you used kvik's clone 12:47 < joe9> mischief: I think I checked it out. but, have not tried it. Would you recommend it over replica(8)? 12:51 < mischief> joe9: i'm not very familiar with how replica works, but clone can avoid most of the latency penalty when copying multiple files over 9p 12:55 < joe9> mischief: ok, thanks. will check it out. 12:56 < joe9> I vaguely recall that it was similar to cp -x though. 13:00 < mischief> joe9: clone uses a process pool to do things in parallel.. cp is serial 13:00 < mischief> time a copy of a file tree off a remote 9p server with dircp or cp and then do it with clone 13:55 < joe9> BurnZeZ: regarding the replica log files, any suggestions on when to purge them? after each transfer? 13:57 < BurnZeZ> The log is used to determine what files to update 13:58 < BurnZeZ> I guess maybe it can be deleted 13:59 < BurnZeZ> But I'm not entirely sure 14:04 < joe9> ok, thanks. Burnzez, I am trying out a little hack. Before doing applylog, I have an awk script do fcp -x of any big files. joe9 : Burnzez, I am trying out a little hack. Before doing applylog, I have an awk script do fcp -x of any big files. BurnZeZ : Same with awk that should give better throughput than plain old applylog joe9 : that should give better throughput than plain old applylog BurnZeZ : replica uses fcp half: BurnZeZ: Aye. oh, it does. I thought it did not. I did not see any worker threads. I have to doublecheck though. joe9 : oh, it does. I thought it did not. I did not see any worker threads. I have to doublecheck though. BurnZeZ : Oh wait no BurnZeZ : The replica scripts do BurnZeZ : And it’s only for copying $serverlog to $clientlog oh, actually I take that back. joe9 : oh, actually I take that back. copy1 of applylog has threads joe9 : copy1 of applylog has threads I mean it has different workers joe9 : I mean it has different workers BurnZeZ : Right oh, so, it should be fine then. sorry for the bother. joe9 : oh, so, it should be fine then. sorry for the bother. BurnZeZ : Yeah, this stuff isn’t documented in the manpage BurnZeZ : I don’t know why they bothered writing the scripts instead of making the manpage more clear