12:20 < joe9> BurnZeZ: do you recall how you found out about replica(8). It is cool. I cannot believe that it is not more popular
12:22 < joe9> anyone has the man books? 9FRONT PROGRAMMER'S MANUAL V2 and V1. I am about to order them, but, want to check if anyone has them and find them useful.
12:24 < mycroftiv> joe9: old plan 9 from bell labs used replica/pull as the mechanism to update the distribution
12:25 -!- Irssi: mycroftiv myrkul 
12:25 < kvik> khm: ok, good to know. my domain and patience with gandi webshit is about to expire, so i'm looking around
12:25 < joe9> mycroftiv: do you use it? any practical experiences to share?
12:25 < joe9> kvik: namesilo?
12:25 < aiju> we specifically dropped replica because it sucked
12:25 < mycroftiv> joe9: i like burnzez rewrite of the support scripts to 'rep'
12:25 < aiju> i think it was very slow
12:25 < aiju> but i forgot what other objections there were
12:26 < aiju> obviously centralised
12:26 < mycroftiv> the value of replica(8) as a tool is independent of whether or not it was well used and administered by labs for distro updates
12:27 < mycroftiv> the replica(1) scripts and config are rather over-elaborate which is why i think it hasnt been used very much by people for personal file tree synchronization
12:27 < kvik> my main objection is it's grossly overcomplicated to setup for what you end up with
12:28 < mycroftiv> thats why i like burnzez rep stuff, but since it is undocumented and exists only in burnzez griddisk, it hasnt taken over the world yet
12:28 < joe9> I have data on a hosted vm, my local machine, backup of the local machine.  I used fcp -x and disk/mkfs to synchronise them. but, disk/mkfs only uses modification time. Hence, I had to have a wrapper to ensure that I am not missing stuff.
12:28 < kvik> i should really implement proper sync for clone. but i've been saying that since the winter
12:28 < joe9> when reading about replica, it seems to do exactly that.
12:29 < joe9> but, I am not sure how it is regarded in practice.
12:29 < mycroftiv> yes syncing files between systems is exactly what replica is intended for
12:29 < kvik> yea, it also just compares timestamp iirc
12:29 < joe9> kvik: replica(8) says that it uses modtime and size.
12:30 < kvik> i guess there's just a lot more buttons to press as opposed to bare mkfs
12:30 < mycroftiv> its little used in practice because setting up the config files is one of those things where everyone reads the manapge and says 'uck, maybe ill deal with this some other time'
12:30 < joe9> mycroftiv: do you know where burnzez's scripts are? replica(1)?
12:30 < mycroftiv> joe9: they are in burnzez griddisk directory on the public grid
12:30 < mycroftiv> ive been trying to get him to document them for weeks but burnzez doesnt work to order
12:31 < joe9> that makes sense. I read the man page and realized that I need a print edition and understand them before I can do much.
12:32 < joe9> mycroftiv: do you know if burnzez has any plans to merge his scripts into 9front? or, does he want to leave them as they are.
12:40 < aiju> hm
12:40 < BurnZeZ> I do not because replica sucks
12:41 < aiju> yeah, i think the knowledge of knowledge ofknowledge of ... thing is spot on
12:44 < joe9> BurnZeZ: if you have a few free minutes, what do you not like about replica? speed? or hard to configure? 
12:45 < joe9> I have a high latency link to my hosted vm. I use fcp -x to transfer files. 
12:45 < BurnZeZ> While speed is not terrible, it's not great
12:45 < BurnZeZ> Configuration difficulty is one factor
12:45 < joe9> I want to replace it with replica(?) 
12:45 < BurnZeZ> The other part is that replica seems to be designed with a parent-child dichotomy
12:45 < BurnZeZ> client-server
12:45 < BurnZeZ> But it doesn't make much sense to do so
12:46 < BurnZeZ> Really it should be bidirectional, and it has a bunch of nonsense to do just that
12:46 < joe9> my use case is that I always get stuff from the hosted vm. so, the client-server is fine here.
12:46 < joe9> and I am always sending stuff to the backup.
12:46 < BurnZeZ> If you do have synchronization needs, it works okay
12:47 < joe9> so, I have no need for bidirectional stuff.
12:47 < BurnZeZ> It breaks with append-only files
12:47 < joe9> oh, ok. I am not using append-only files.
12:47 < joe9> I had issues with append-only files too.
12:48 < joe9> even fcp or cp (I cannot recall) messed such files up.

burnzez. do you know if the rep stuff is hosted somewhere else other than on griddisk?
joe9 : burnzez. do you know if the rep stuff is hosted somewhere else other than on griddisk?
BurnZeZ :  Probably not
I am not sure what the purpose of the log file is. in /n/griddisk/burnzez/rep/rep. But, it lists some files which are not in root/rc/bin/rep
joe9 : I am not sure what the purpose of the log file is. in /n/griddisk/burnzez/rep/rep. But, it lists some files which are not in root/rc/bin/rep
kvik : wewantrepdocs.com is free
stillneedrepdocs : i had a few burnzez lines saved at one point i copy/pasted to explain things so i could probably grep the logs for them
stillneedrepdocs : rep grep
BurnZeZ :  nuts
BurnZeZ :  I was going to write docs then got super demotivated because I realized replica’s problems
are there any other problems?
joe9 : are there any other problems?
stillneedrepdocs : moody : BurnZeZ: do you have a quick reference for rep ?
stillneedrepdocs : mycroftiv : BurnZeZ : mkdir /rc/bin/rep; cp /n/griddisk/burnzez/rep/rep/root/rc/bin/rep/* /rc/bin/rep/
stillneedrepdocs : mycroftiv : BurnZeZ : mkdir /usr/$user/lib/rep
stillneedrepdocs : mycroftiv : BurnZeZ : rep/cinit /n/griddisk/burnzez/rep/libautism/ libautism
stillneedrepdocs : mycroftiv : BurnZeZ : This line creates the directory /usr/$user/lib/rep/libautism
stillneedrepdocs : mycroftiv : BurnZeZ : rep/pull libautism
BurnZeZ :  I don’t recall any at the moment
mycroftiv : if i recall, you probably also want to rep/cinit for rep itself too

10:33 < joe9> BurnZeZ: I am familiarising myself with replica code. I have to profile it and figure out why it is slow. From a cursory reading, I would assume it is due to finddb(?).
10:33 < BurnZeZ> Hmm
10:34 < BurnZeZ> Replica shouldn't be slow
10:34 < joe9> on a local machine, I can see it going through each file. 
10:34 < BurnZeZ> Ah
10:34 < joe9> replca/applylog is what is being run when I do replica/pull -nv
10:34 < BurnZeZ> You're using the replica scripts
10:34 < joe9> this is for the first time that replica runs.
10:34 < joe9> yes, I am using replica's scripts
10:35 < BurnZeZ> I wrote rep because I felt the original scripts were too complicated
10:35 < joe9> not yours yet. I am want to understand what was before before I can figure out your changes.
10:35 < joe9> oh, ok.
10:35 < BurnZeZ> It's probably worth learning replica(8) instead of the scripts
10:35 < BurnZeZ> Err
10:35 < joe9> rep is a replacement of replica(1)?
10:35 < BurnZeZ> Yeah, replica(8)
10:35 < BurnZeZ> joe9: Basically
10:36 < BurnZeZ> There is two things
10:36 < BurnZeZ> replica, and replica rc scripts
10:36 < joe9> but, it is not replica(1) that is the issue. I can see the replica/applylog going through each file.
10:36 < BurnZeZ> Yeah
10:36 < joe9> replica/applylog belongs to replica(8)
10:37 < BurnZeZ> Another bad thing
10:37 < joe9> to understand, rep is a wrapper around replica(8), correct?
10:37 < BurnZeZ> replica(8) is not fully documented
10:37 < BurnZeZ> There is undocumented parameters that are important
10:37 < joe9> the code looks in cinap's style. Did cinap write replica/applylog?
10:38 < BurnZeZ> It's bell labs stuff I think
10:38 < joe9> 10:37 < joe9> to understand, rep is a wrapper around replica(8), correct?
10:38 < joe9> would you mind answering it, please?
10:38 < BurnZeZ> joe9: One thing replica/applylog does that sucks
10:39 < BurnZeZ> Yes, rep is scripts that wraps replica(8)
10:39 < BurnZeZ> it spools files to /tmp/ first
10:39 < BurnZeZ> Which slows everything down
10:39 < Aram> aiju: ugh, I didn't know such things existed at all: https://mathoverflow.net/questions/43462/existence-of-a-smooth-function-with-nowhere-converging-taylor-series-at-every-po
10:40 < BurnZeZ> joe9: It may be worth updating replica(8) so that the manpage and code are synchronized
10:40 < joe9> it spools files to /tmp/ first - thanks, I see the big files in /tmp/. I cannot figure out what they are though.
10:40 < joe9> replica09550629 is the filename
10:40 < BurnZeZ> Yeah
10:40 < BurnZeZ> It's really stupid
10:40 < Aram> in fact
10:40 < Aram> https://en.wikipedia.org/wiki/Non-analytic_smooth_function
10:40 < BurnZeZ> I mean, I understand why it is done
10:41 < BurnZeZ> And it has a good reason
10:41 < BurnZeZ> But not good enough
10:41 < joe9> which file is it? Is it the original file from the server?
10:41 < BurnZeZ> Yes
10:41 < joe9> s/which/what/
10:41 < BurnZeZ> It copies the server file to /tmp/ before replacing the client file with it
10:41 < joe9> oh, really. over a low latency link, that would be a killer
10:42 < joe9> even if the files are the same?
10:42 < BurnZeZ> No
10:42 < BurnZeZ> However
10:42 < BurnZeZ> It does not do delta synchronization
10:42 < joe9> I understand and do not want delta sync
10:42 < BurnZeZ> Just making sure that's clear
10:43 < BurnZeZ> If the server file is modified, the whole file must be copied
10:43 < BurnZeZ> And it copies to /tmp/ first
10:43 < BurnZeZ> I think the idea is so you can be sure you could actually GET the entire file
10:43 < BurnZeZ> Before you replace the client file with it
10:43 < BurnZeZ> Instead of start replacing the file and then some read error
10:44 < joe9> yes, that makes sense.
10:44 < joe9> In my case, this is the first time I am running replica and it should not be doing it when the files are the same.
10:44 < joe9> let me reset the client db and log and try.
10:44 < BurnZeZ> Hmm
10:44 < BurnZeZ> Yeah, it sounds like the database is not right
10:44 < BurnZeZ> That's part of why I wrote rep
10:44 < BurnZeZ> But after I wrote it, now I hate replica :)
10:45 < BurnZeZ> The best thing would be to use venti for all this
10:45 < BurnZeZ> Then we would even have delta synchronization
10:45 < BurnZeZ> For free
10:45 < BurnZeZ> venti concept, I mean
10:46 < BurnZeZ> Not the implementation that exists
10:46 < joe9> another thing that I cannot understand, When I start from scratch, no db or log file, I get this error
10:46 < joe9> replica/compactdb: opendb /dist/replica/client/ddfbkp.db: '/dist/replica/client/ddfbkp.db' does not exist
10:47 < joe9> this is probably replica(1) in play(?)
10:47 < BurnZeZ> touch the file first
10:47 < BurnZeZ> I don't think the scripts will create it
10:47 < BurnZeZ> I could be wrong
10:48 < joe9> yes, it works then. But, in that scenario, it skips the !havedb if statement in applylog
10:48 < joe9> I am wrong.
10:48 < BurnZeZ> Ah, right
10:48 < BurnZeZ> Oh
10:48 < joe9> sorry, my mistake, I think.
10:48 < BurnZeZ> Yeah, the scripts are confusing
10:48 < BurnZeZ> I recommend ignoring them and using replica(8) directly
10:49 < joe9> ok, thanks. will think this through. 
10:49 < BurnZeZ> 9fs 9contrib && replica/updatedb -r /n/9contrib/contrib $home/lib/replica/contrib/sdb >> $home/lib/replica/contrib/log
10:49 < BurnZeZ> 9fs 9contrib && replica/applylog $home/lib/replica/contrib/cdb /tmp/contrib/ /n/9contrib/contrib < $home/lib/replica/contrib/log
10:49 < BurnZeZ> A couple old snippets
10:49 < yuuko> https://kurofuku.me/i/1559843282.png
10:49 < joe9> yes, please, these will help me a lot.
10:49 < BurnZeZ> This doesn't use replica(1)
10:49 < BurnZeZ> Even though it uses the same directory paths
10:50 < BurnZeZ> I think
10:51 < joe9> ok, thanks.
10:51 < BurnZeZ> I really dislike the replica scripts
10:51 < BurnZeZ> They are just way too complicated
10:52 < BurnZeZ> If it was up to me, I would get rid of them entirely
10:52 < joe9> ok, thanks. I will read up on your scripts now that I have a clue.
10:52 < BurnZeZ> All they do is confuse people who want to use replica
10:54 < BurnZeZ> ftpfs -a my@mail goes.gsfc.nasa.gov && replica/updatedb -r /n/ftp/pub/goescolor/goeseast/hurricane2/color_lrg -x older_images /tmp/replica/goes.gsfc.nasa.gov/sdb >> /tmp/replica/goes.gsfc.nasa.gov/log
10:54 < BurnZeZ> ftpfs -a my@mail goes.gsfc.nasa.gov && replica/applylog /tmp/replica/goes.gsfc.nasa.gov/cdb (/tmp/ftp/goes.gsfc.nasa.gov /n/ftp)^/pub/goescolor/goeseast/hurricane2/color_lrg/ < /tmp/replica/goes.gsfc.nasa.gov/log
10:54 < BurnZeZ> I doubt these will be valid anymore
10:54 < BurnZeZ> But it's another interesting usage
11:03 < joe9> to first seed the client db, is there a replica/scan on the client?
11:03 < joe9> s/is there a/is there a way to do/
11:04 < BurnZeZ> You can run updatedb on the client root, and create the client database
11:04 < joe9> ok, that should help me. Thanks.
11:05 < BurnZeZ> But I'm not sure if this will confuse replica in some way
11:05 < BurnZeZ> Ideally it should work
11:05 < joe9> ok, thanks. I should focus on replica(8) then. Thanks.
11:33 < joe9> BurnZeZ: do you recall what this does? clientdb clientroot serverroot [ path ... ]
11:33 < joe9> it is documented twice in replica(8) too.
11:33 < joe9> I mean the command.
11:33 < BurnZeZ> It's the other commands being wrapped
11:34 < BurnZeZ> man -P 8 replica
12:33 < mischief> joe9: have you used kvik's clone
12:47 < joe9> mischief: I think I checked it out. but, have not tried it. Would you recommend it over replica(8)?
12:51 < mischief> joe9: i'm not very familiar with how replica works, but clone can avoid most of the latency penalty when copying multiple files over 9p
12:55 < joe9> mischief: ok, thanks. will check it out. 
12:56 < joe9> I vaguely recall that it was similar to cp -x though.
13:00 < mischief> joe9: clone uses a process pool to do things in parallel.. cp is serial
13:00 < mischief> time a copy of a file tree off a remote 9p server with dircp or cp and then do it with clone
13:55 < joe9> BurnZeZ: regarding the replica log files, any suggestions on when to purge them? after each transfer?
13:57 < BurnZeZ> The log is used to determine what files to update
13:58 < BurnZeZ> I guess maybe it can be deleted
13:59 < BurnZeZ> But I'm not entirely sure
14:04 < joe9> ok, thanks.

Burnzez, I am trying out a little hack. Before doing applylog, I have an awk script do fcp -x of any big files.
joe9 : Burnzez, I am trying out a little hack. Before doing applylog, I have an awk script do fcp -x of any big files.
BurnZeZ :  Same with awk
that should give better throughput than plain old applylog
joe9 : that should give better throughput than plain old applylog
BurnZeZ :  replica uses fcp
half: BurnZeZ: Aye.
oh, it does. I thought it did not. I did not see any worker threads. I have to doublecheck though.
joe9 : oh, it does. I thought it did not. I did not see any worker threads. I have to doublecheck though.
BurnZeZ :  Oh wait no
BurnZeZ :  The replica scripts do
BurnZeZ :  And it’s only for copying $serverlog to $clientlog
oh, actually I take that back.
joe9 : oh, actually I take that back.
copy1 of applylog has threads
joe9 : copy1 of applylog has threads
I mean it has different workers
joe9 : I mean it has different workers
BurnZeZ :  Right
oh, so, it should be fine then. sorry for the bother.
joe9 : oh, so, it should be fine then. sorry for the bother.
BurnZeZ :  Yeah, this stuff isn’t documented in the manpage
BurnZeZ :  I don’t know why they bothered writing the scripts instead of making the manpage more clear