Obsidian does have support for block references though, it just doesn't create them by default (i.e. always). So I wonder if there is a way to sort of simulate the burden that the other apps are dealing with by modifying your .md test files *just* for Obsidian to include block references on every, er, "paragraph" I guess it'd be. You know what I mean? 🤔
Obsidian actually does internally track every "block" in every file, along with list items and tasks and headings and links and whatnot. All are indexed live as you update files. You can see this if you type `[[some note#^^` and you will get suggestions for which paragraph you want to link to.
That's true. But is that the same thing as what Roam, etc. are doing? There does appear to be a difference in Obsidian between being able to find some piece of text with e.g. a header or paragraph link search, vs. the actual block ID that gets assigned if you complete the link. I'm not confident there is a performance difference, but there is at least some kind of functional difference. So that's what I'm curious about testing, though I'm unsure whether it would be worth the effort of adding the block ref creator in the tools used to generate these files.
I don't know what you mean by block ref creator, but Obsidian will accept any unique-within-the-file alphanumeric string as a block ID, you can literally generate them at random or just use ^1, ^2, ^3 as far as I know.
As I said, though, Obsidian does actually index every block *and* heading in every file, in the sense that it has in-memory data for "a markdown block of this type exists at this position/length within this file, and (optionally) has this ID". Thus, if you link to a paragraph, list item, or other block with an ID, it jumps directly there without any searching, and can also embed (transclude) the block in another note.
Heading links are also indexed, which is why Obsidian can know (again, without reading or searching the file) what range of characters in the file corresponds to "the section under the heading of the given name". These are some of the things that make it so fast, e.g. for hover-previewing links to such sections or blocks -- it doesn't have to parse and render all the markdown in the file, just the referenced part.
There are some small aspects of what you're saying that don't quite align with my understanding. However broadly it seems you are correct. It is just a bit hard to believe that Obsidian and e.g. Roam, Logseq (local as well), etc. are truly doing the same work, and Obsidian is just *that much* faster. Part of the reason I say this is that Obsidian is not considered a "true" outliner, even with the outliner plugin(s).
Whether this is simply a misinformed opinion, or based more on semantic argument (Obsidian is not a *strict* outliner, therefore not a "real" one), I don't know. But it has been strongly suggested that those who favor outliners miss certain features in Obsidian that "real" outliners have, and my understanding was that often centered around block-based functions. Are there not real differences between how Obsidian and Roam actually work?
That said I'm not as knowledgeable as you seem to be about the inner workings/behind the scenes. I'm more going off of the comments of others previously, and comparisons others have done between e.g. Obsidian and Roam. I'm an Obsidian user, I couldn't deal with the restrictions of outliners, so I'm definitely not advocating for Roam or any other tool over Obsidian. 😄 Just wanting to make sure that the benchmark results for Obs are "fair" and *truly* comparable. If they are, it's very impressive indeed.
I think the public perception of Obsidian by Roamans is based on a combination of outdated information (some of the block indexing I'm describing has only existed for about 9 months) and the lack of highly-visible UI affordances for some (well, most) of the functionality.
For example, Obsidian has always had swap line up/down commands, and indent/outdent commands, not to mention fold-toggling commands that would, in combination, allow you to use it as an outliner.
But the swap up/down commands don't have a default hotkey assignment, so you have to assign them yourself and know they exist in the first place, and you'd also have to know to enable the "fold indent" setting that lets lists expand and contract. (There's also an outliner plugin that gives you even better ability to move things around without first selecting all the lines you want to move, but it's also a relatively recent development.)
Similarly, though Obsidian has more-or-less always had the ability to link and embed blocks and track them in back references, there are many small creature comforts like showing reference counts inline or easily creating a block ID from the block (as opposed to linking to the block from elsewhere) that require a plugin to get a nice UI affordance.
So, if you wanted to capture all the same functionality at the UI level, you would probably need to install the Outliner, Copy Block Link, and perhaps the Zoom plugin as well, that lets you focus on a small part of a note.
(That being said, I would be surprised if adding any of those plugins substantially affected any of your benchmarks aside from startup time, as they mainly provide additional commands, menu items, and hotkeys in the editor.)
Given that the block and outline functions of the core app are basically hidden unless you know which settings to turn on and what commands to add hotkeys for, and that the only way by default to create a block link or embed is the autocompletion when you type (and you have to know to type `[[note name#^^` to trigger it), it's 100% understandable someone could walk away with the impression Obsidian lacks the same outlining and block reference capabilities as Roam. And in truth, to get something close to the "full" experience, you do need the plugins I mentioned. So it is in some sense fair to say "it's not the same" because of this additional setup/learning curve.
However, the *technical* argument that "Obsidian's not the same because it's not a database" is just flat wrong. Obsidian does have a database, it's in IndexedDB, and it indexes every paragraph and every line of every list along with block IDs and the hierarchies thereof, and it's almost-instantly updated whenever the notes change, even if you edit them in another program! Which gives Obsidian a level of ability to integrate with other tools that's unmatched by the competition.
There are probably other misconceptions as well, such as it not being immediately obvious that Obsidian updates links when you rename files, or that you can have it update links to sections as well by using the "rename this heading" command. There are *tons* of tools and commands like that, that you'll be unlikely to discover in the first hour or two of use, and/or were added in the last six months. (If you aren't involved in the community or follow the releases, you won't find out a lot about this stuff.)
Anyway, to answer your question about whether there are "real differences" between how Obsidian and Roam work, well absolutely there are. Roam doesn't have note files stored as markdown, after all. Roam is DB-centric, Obsidian is text-centric.
That is, Obsidian creates its database as a background-updated eventually-consistent index of the text. This is what makes it faster: since the text is primary, it is available and editable right away. While you're using it, it updates its index in the background, so you are never waiting on the database to do something, it just takes a second or two before secondary views (like tags, backlinks, outline, etc.) will be caught up with whatever you typed.
(And you generally won't notice that because if you're typing you're not looking at things that show you what you changed, you'll be looking at those things later when you've moved to a different note!)
Thus, you could say that Obsidian gets its performance by cheating: it's cleverly designed so that nearly all the hard work takes place at times and in places where you can't benchmark it doing so. ;-)
Finally, the inner workings I've been describing are actually documented on Github, at https://github.com/obsidianmd/obsidian-api/blob/master/obsidian.d.ts -- the MetadataCache and its CachedMetadata records describe precisely what information is indexed and how, so that plugins can use it.
Awesome, thanks! I sense perhaps in your reply that you are (I hope) addressing the broader misconceptions about Obsidian and not just me, specifically. As I mentioned I use Obsidian (for about a year now), and in case it wasn't clear I'm a *huge* fan of it. I left Roam for Obsidian that year ago (with some brief flirtations with Logseq at the time), and overall I'm pretty satisfied with Obsidian for what it is.
I just really wanted to make sure that the performance results here were representative and that when they come up in the general TfT discussion there are not immediately dismissed by appeals to "it's not doing the same work!". Hopefully, thanks to your in-depth response, everyone now understands to what degree it is the same, and in what smaller ways it is different.
Based on your info here it really does sound like Obsidian is just a better-designed approach, at least *for today's capabilities*. I suspect Roam and/or Conor might argue that some hypothetical future meta-capability will be better enabled by the pure database back end vs. markdown-to-DB. But for doing (solo) work today Obsidian definitely presents a great experience!
I have 20,000 notes with lots of attachments and have found that renaming the top level folder breaks obsidian. It also cannot use the knowledge graph without just breaking. I have to force quit and then it has to re-index everything.
Obsidian does have support for block references though, it just doesn't create them by default (i.e. always). So I wonder if there is a way to sort of simulate the burden that the other apps are dealing with by modifying your .md test files *just* for Obsidian to include block references on every, er, "paragraph" I guess it'd be. You know what I mean? 🤔
Obsidian actually does internally track every "block" in every file, along with list items and tasks and headings and links and whatnot. All are indexed live as you update files. You can see this if you type `[[some note#^^` and you will get suggestions for which paragraph you want to link to.
That's true. But is that the same thing as what Roam, etc. are doing? There does appear to be a difference in Obsidian between being able to find some piece of text with e.g. a header or paragraph link search, vs. the actual block ID that gets assigned if you complete the link. I'm not confident there is a performance difference, but there is at least some kind of functional difference. So that's what I'm curious about testing, though I'm unsure whether it would be worth the effort of adding the block ref creator in the tools used to generate these files.
I don't know what you mean by block ref creator, but Obsidian will accept any unique-within-the-file alphanumeric string as a block ID, you can literally generate them at random or just use ^1, ^2, ^3 as far as I know.
As I said, though, Obsidian does actually index every block *and* heading in every file, in the sense that it has in-memory data for "a markdown block of this type exists at this position/length within this file, and (optionally) has this ID". Thus, if you link to a paragraph, list item, or other block with an ID, it jumps directly there without any searching, and can also embed (transclude) the block in another note.
Heading links are also indexed, which is why Obsidian can know (again, without reading or searching the file) what range of characters in the file corresponds to "the section under the heading of the given name". These are some of the things that make it so fast, e.g. for hover-previewing links to such sections or blocks -- it doesn't have to parse and render all the markdown in the file, just the referenced part.
There are some small aspects of what you're saying that don't quite align with my understanding. However broadly it seems you are correct. It is just a bit hard to believe that Obsidian and e.g. Roam, Logseq (local as well), etc. are truly doing the same work, and Obsidian is just *that much* faster. Part of the reason I say this is that Obsidian is not considered a "true" outliner, even with the outliner plugin(s).
Whether this is simply a misinformed opinion, or based more on semantic argument (Obsidian is not a *strict* outliner, therefore not a "real" one), I don't know. But it has been strongly suggested that those who favor outliners miss certain features in Obsidian that "real" outliners have, and my understanding was that often centered around block-based functions. Are there not real differences between how Obsidian and Roam actually work?
That said I'm not as knowledgeable as you seem to be about the inner workings/behind the scenes. I'm more going off of the comments of others previously, and comparisons others have done between e.g. Obsidian and Roam. I'm an Obsidian user, I couldn't deal with the restrictions of outliners, so I'm definitely not advocating for Roam or any other tool over Obsidian. 😄 Just wanting to make sure that the benchmark results for Obs are "fair" and *truly* comparable. If they are, it's very impressive indeed.
I think the public perception of Obsidian by Roamans is based on a combination of outdated information (some of the block indexing I'm describing has only existed for about 9 months) and the lack of highly-visible UI affordances for some (well, most) of the functionality.
For example, Obsidian has always had swap line up/down commands, and indent/outdent commands, not to mention fold-toggling commands that would, in combination, allow you to use it as an outliner.
But the swap up/down commands don't have a default hotkey assignment, so you have to assign them yourself and know they exist in the first place, and you'd also have to know to enable the "fold indent" setting that lets lists expand and contract. (There's also an outliner plugin that gives you even better ability to move things around without first selecting all the lines you want to move, but it's also a relatively recent development.)
Similarly, though Obsidian has more-or-less always had the ability to link and embed blocks and track them in back references, there are many small creature comforts like showing reference counts inline or easily creating a block ID from the block (as opposed to linking to the block from elsewhere) that require a plugin to get a nice UI affordance.
So, if you wanted to capture all the same functionality at the UI level, you would probably need to install the Outliner, Copy Block Link, and perhaps the Zoom plugin as well, that lets you focus on a small part of a note.
(That being said, I would be surprised if adding any of those plugins substantially affected any of your benchmarks aside from startup time, as they mainly provide additional commands, menu items, and hotkeys in the editor.)
Given that the block and outline functions of the core app are basically hidden unless you know which settings to turn on and what commands to add hotkeys for, and that the only way by default to create a block link or embed is the autocompletion when you type (and you have to know to type `[[note name#^^` to trigger it), it's 100% understandable someone could walk away with the impression Obsidian lacks the same outlining and block reference capabilities as Roam. And in truth, to get something close to the "full" experience, you do need the plugins I mentioned. So it is in some sense fair to say "it's not the same" because of this additional setup/learning curve.
However, the *technical* argument that "Obsidian's not the same because it's not a database" is just flat wrong. Obsidian does have a database, it's in IndexedDB, and it indexes every paragraph and every line of every list along with block IDs and the hierarchies thereof, and it's almost-instantly updated whenever the notes change, even if you edit them in another program! Which gives Obsidian a level of ability to integrate with other tools that's unmatched by the competition.
There are probably other misconceptions as well, such as it not being immediately obvious that Obsidian updates links when you rename files, or that you can have it update links to sections as well by using the "rename this heading" command. There are *tons* of tools and commands like that, that you'll be unlikely to discover in the first hour or two of use, and/or were added in the last six months. (If you aren't involved in the community or follow the releases, you won't find out a lot about this stuff.)
Anyway, to answer your question about whether there are "real differences" between how Obsidian and Roam work, well absolutely there are. Roam doesn't have note files stored as markdown, after all. Roam is DB-centric, Obsidian is text-centric.
That is, Obsidian creates its database as a background-updated eventually-consistent index of the text. This is what makes it faster: since the text is primary, it is available and editable right away. While you're using it, it updates its index in the background, so you are never waiting on the database to do something, it just takes a second or two before secondary views (like tags, backlinks, outline, etc.) will be caught up with whatever you typed.
(And you generally won't notice that because if you're typing you're not looking at things that show you what you changed, you'll be looking at those things later when you've moved to a different note!)
Thus, you could say that Obsidian gets its performance by cheating: it's cleverly designed so that nearly all the hard work takes place at times and in places where you can't benchmark it doing so. ;-)
Finally, the inner workings I've been describing are actually documented on Github, at https://github.com/obsidianmd/obsidian-api/blob/master/obsidian.d.ts -- the MetadataCache and its CachedMetadata records describe precisely what information is indexed and how, so that plugins can use it.
Thank you for this awesome in-depth comment, much appreciated.
Awesome, thanks! I sense perhaps in your reply that you are (I hope) addressing the broader misconceptions about Obsidian and not just me, specifically. As I mentioned I use Obsidian (for about a year now), and in case it wasn't clear I'm a *huge* fan of it. I left Roam for Obsidian that year ago (with some brief flirtations with Logseq at the time), and overall I'm pretty satisfied with Obsidian for what it is.
I just really wanted to make sure that the performance results here were representative and that when they come up in the general TfT discussion there are not immediately dismissed by appeals to "it's not doing the same work!". Hopefully, thanks to your in-depth response, everyone now understands to what degree it is the same, and in what smaller ways it is different.
Based on your info here it really does sound like Obsidian is just a better-designed approach, at least *for today's capabilities*. I suspect Roam and/or Conor might argue that some hypothetical future meta-capability will be better enabled by the pure database back end vs. markdown-to-DB. But for doing (solo) work today Obsidian definitely presents a great experience!
I have 20,000 notes with lots of attachments and have found that renaming the top level folder breaks obsidian. It also cannot use the knowledge graph without just breaking. I have to force quit and then it has to re-index everything.
I'm already an adoring user of Obsidian, but reading this and the above comments made me even happier. Thanks!