The Magic of Being a Programmer: RFC 499 Added

About 4 hours ago, I posted that I have just had uploaded the 100th RFC after a week or more uploading the RFC documents manually. I also said that this rate is not acceptable if we want to upload 5000+ RFCs. Well, 5 minutes ago, 4 hours after that post, I’ve just finished uploading the 499th RFC document!

What happened?

3 hours of coding in MS Visual Studio and an hour of importing XML pages into the wiki is exactly what happened!

I exported some pages to XML from the wiki, looked at the XML output, written a program in C# that will allow me to add RFC files to a list of RFC titles to import into the wiki then allows me to export these text files into an XML format that is readable by the MediWiki software on the site. I started exporting 10 pages or so, fixed some problems with the code (mostly formatting RFC text into valid XML text) and finally it started to work. I’ve uploaded about 400 RFCs so far using that small program and it has been working fine.

Of course it won’t work for anyone who does not have admin right on the wiki, but the concept is the same for all MediWiki based wikis. So, when I create a page to describe the program and upload it’s code, all you’ll have to do is to change some values in an XML config file, name your text files so that the file name equals the title you want for it when it is uploaded (I had to create another small program to rename the RFC text files to become appropriate to be used as titles in kDemons’ wiki), add the text files you want to add and export them to XML, then finally, import them from your Special:Import page into the wiki, and you have all your text files as articles in your wiki then!

The only problem is you’ll have to wait for me to upload the program and its code (It is 3:00

AM around here right now, so I have to at least get some sleep!)

However, here’s a copy-paste version of the code (that’s the best I could do for now!)

RFC Importer Form in Design View in Visual Studio 2005

MainForm.cs — the code file for the main form

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.Xml;
using System.IO;
using kDemons.Windows.RFCImporter.Properties;

namespace kDemons.Windows.RFCImporter {
public partial class MainForm : Form {
private string fileFormat = “”;
private string pageFormat = “”;
private long pageId = Settings.Default.StartPageId – 1;
private long revId = Settings.Default.StartRevisionId – 1;

public MainForm() {
InitializeComponent();
StringBuilder builder = new StringBuilder(1084);
builder.Append(”<mediawiki xmlns=\”http://www.mediawiki.org/xml/export-0.3/\” xmlns:xsi=\”http://www.w3.org/2001/XMLSchema-instance\” xsi:schemaLocation=\”http://www.mediawiki.org/xml/export-0.3/ http://www.mediawiki.org/xml/export-0.3.xsd\” version=\”0.3\” xml:lang=\”en\”>\n”);
builder.Append(”<siteinfo>\n”);
builder.Append(”<sitename>Knowledge Demons</sitename>\n”);
builder.Append(”<base>http://www.kdemons.com/index.php/Main_Page</base>\n”);
builder.Append(”<generator>MediaWiki 1.15.0</generator>\n”);
builder.Append(”<case>first-letter</case>\n”);
builder.Append(”<namespaces>\n”);
builder.Append(”<namespace key=\”-2\”>Media</namespace>\n”);
builder.Append(”<namespace key=\”-1\”>Special</namespace>\n”);
builder.Append(”<namespace key=\”0\”/>\n”);
builder.Append(”<namespace key=\”1\”>Talk</namespace>\n”);
builder.Append(”<namespace key=\”2\”>User</namespace>\n”);
builder.Append(”<namespace key=\”3\”>User talk</namespace>\n”);
builder.Append(”<namespace key=\”4\”>kDemons</namespace>\n”);
builder.Append(”<namespace key=\”5\”>kDemons talk</namespace>\n”);
builder.Append(”<namespace key=\”6\”>File</namespace>\n”);
builder.Append(”<namespace key=\”7\”>File talk</namespace>\n”);
builder.Append(”<namespace key=\”8\”>MediaWiki</namespace>\n”);
builder.Append(”<namespace key=\”9\”>MediaWiki talk</namespace>\n”);
builder.Append(”<namespace key=\”10\”>Template</namespace>\n”);
builder.Append(”<namespace key=\”11\”>Template talk</namespace>\n”);
builder.Append(”<namespace key=\”12\”>Help</namespace>\n”);
builder.Append(”<namespace key=\”13\”>Help talk</namespace>\n”);
builder.Append(”<namespace key=\”14\”>Category</namespace>\n”);
builder.Append(”<namespace key=\”15\”>Category talk</namespace>\n”);
builder.Append(”</namespaces>\n”);
builder.Append(”</siteinfo>\n”);
builder.Append(”{0}\n”);
builder.Append(”</mediawiki>”);
fileFormat = builder.ToString();

builder = new StringBuilder(264);
builder.Append(”<page>\n”);
builder.Append(”<title>{0}</title>\n”);
builder.Append(”<id>{1}</id>\n”);
builder.Append(”<revision>\n<id>{2}</id>\n”);
builder.Append(”<timestamp>{3}-{4}-{5}T{6}:{7}:{8}Z</timestamp>\n”);
builder.Append(”<contributor>\n<username>{9}</username>\n<id>{10}</id>\n</contributor>\n”);
builder.Append(”<comment>{11}</comment>\n”);
builder.Append(”<text xml:space=\”preserve\”>\n”);
builder.Append(”&lt;pre&gt;\n{12}\n&lt;/pre&gt;\n[[Category:RFC]]\n”);
builder.Append(”</text>\n</revision>\n</page>\n”);
pageFormat = builder.ToString();
}
private void btnAddRFCs_Click(object sender, EventArgs e) {
if (dlgAddRFCText.ShowDialog() == DialogResult.OK) {
int i = 0;
foreach (string file in dlgAddRFCText.FileNames) {
if (!lstRFCs.Items.Contains(file)) {
lstRFCs.Items.Add(file); i++;
}
}
if (i > 0) { MessageBox.Show(i.ToString() + ” RFC files added!”); }
lblItemsCount.Text = lstRFCs.Items.Count.ToString();
btnRemoveRFCs.Enabled = btnExport.Enabled = (lstRFCs.Items.Count > 0);
}
}
private void btnRemoveRFCs_Click(object sender, EventArgs e) {
if (lstRFCs.SelectedItems != null && lstRFCs.SelectedItems.Count > 0) {
int i = 0, max = lstRFCs.SelectedItems.Count;
for (int j = 0; j < max; j++) {
lstRFCs.Items.Remove(lstRFCs.SelectedItems[0]); i++;
}
if (i > 0) { MessageBox.Show(i.ToString() + ” RFC files removed!”); }
lblItemsCount.Text = lstRFCs.Items.Count.ToString();
btnRemoveRFCs.Enabled = btnExport.Enabled = (lstRFCs.Items.Count > 0);
}
}

private void btnExport_Click(object sender, EventArgs e) {
if (dlgSaveXMLFile.ShowDialog() == DialogResult.OK) {
try {
StringBuilder pages = new StringBuilder(50000);
foreach (object f in lstRFCs.Items) {
StreamReader reader = new StreamReader(f.ToString());
string rfc = reader.ReadToEnd(); reader.Close(); reader.Dispose();
rfc = rfc.Replace(”“, “”);
rfc = rfc.Replace(””, “”);
rfc = rfc.Replace(”&”, “&amp;”);
rfc = rfc.Replace(”<”, “&lt;”);
rfc = rfc.Replace(”>”, “&gt;”);
rfc = rfc.Replace(”\”", “&quot;”);
pages.Append(makePage(rfc, f.ToString()));
}
string contents = string.Format(fileFormat, pages.ToString());
StreamWriter writer = new StreamWriter(dlgSaveXMLFile.FileName);
writer.Write(contents);
writer.Flush();
writer.Close();
MessageBox.Show(lstRFCs.Items.Count.ToString() + ” RFCs were imported successfully!”);
}
catch (Exception ex) {
MessageBox.Show(ex.Message);
}
}
}

private string makePage(string rfcContent, string rfcFileName) {
pageId++; revId++;
return string.Format(pageFormat, Path.GetFileNameWithoutExtension(rfcFileName),
pageId, revId, DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day,
DateTime.Now.Hour, DateTime.Now.Minute, DateTime.Now.Second,
Settings.Default.AuthorUserName, Settings.Default.AuthorId,
Settings.Default.SummaryComment, rfcContent);
}
}
}

application config file

<?xml version=”1.0″ encoding=”utf-8″ ?>
<configuration>
<configSections>
<sectionGroup name=”applicationSettings” type=”System.Configuration.ApplicationSettingsGroup, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ >
<section name=”kDemons.Windows.RFCImporter.Properties.Settings” type=”System.Configuration.ClientSettingsSection, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ requirePermission=”false” />
</sectionGroup>
</configSections>
<applicationSettings>
<kDemons.Windows.RFCImporter.Properties.Settings>
<setting name=”StartPageId” serializeAs=”String”>
<value>5000</value>
</setting>
<setting name=”StartRevisionId” serializeAs=”String”>
<value>5000</value>
</setting>
<setting name=”AuthorUserName” serializeAs=”String”>
<value>VC</value>
</setting>
<setting name=”AuthorId” serializeAs=”String”>
<value>2</value>
</setting>
<setting name=”SummaryComment” serializeAs=”String”>
<value>Initial import of RFC document Contents</value>
</setting>
</kDemons.Windows.RFCImporter.Properties.Settings>
</applicationSettings>
</configuration>

2 minutes ago, I finally uploaded the RFC 100 document to kDemons.com’s wiki!

Only around 5000+ remaining.

At this rate, it will only require another 500-700 days to complete the whole library, which by all means is not acceptable.

The good news is that I’m only uploading the documents manually while I develop a small program to export the text files into XML that can be imported to the site in bulk! The problem is that I’m not finding the time to do so (not only the real time, but time when you can write code!)

If and when the program is completed, I’ll publish it as open source code and at the same time it will most probably be a good sample of file system and XML handling in .NET (C#)

I’ll post more news when there are more news!

Finally, a page that shows the site map of the kDemons wiki section was created. You can view the sitemap at: http://www.kdemons.com/index.php/kDemons:Sitemap

It’s not much, but it is a start for stabilizing the categorization for now at what it is! It will be hard to add to the current form of the sitemap when new categories are added, but not impossible. However, because editing the sitemap is a complex sensitive task, the page was protected so that only experienced editors can edit it (hoping that even experienced editors won’t make a miss of it as it is right now!

I’ll be creating a more brief template for the sitemap to be included in other pages and templates when I have the time.

The first project of kDemons to start implementing and that is actually related to kDemons’ contents is finally here!

I have started a project to import all RFC documents (the defining documents of the internet) so that they can be published yet for another time. However, with the wiki capabilities of the site, I hope that more than just republishing the RFCs is going to be happening.

For experienced users and contributors who already understand the RFCs or some of them, we can start writing some explanations to accompany the RFCs themselves. The formatting of the RFCs can be fancied up a little and finaly, graphs and the like could be created for the documents.

The project aims at re-publishing a “pretty”, readable and understandable version of the RFCs. If this succeeds, all contents will be initially imported from the RFC Editor site as plain text document added into <pre> tags in kDemons pages. Each RFC will be imported into one kDemons page. Currently, if anyone would like to add some comments explaining the RFC, it should be added to a sub-page of the RFC’s page named “About”.

The category for all RFC documents is RFC. Still working on some templates to add description and meta data to resources in general and RFCs in specific, I’l post when something new comes up in that area. Salute for now!

N.B. I’ve contacted the RFC Editor site asking about the license of their downloads to check for any future problems that might arise and to ask for permission to redistribute the documents. I’m still waiting for their response, and I’ll update this post when one arrives!

• Regarding categorization, a templates category tree has been created to allow categorization of templates. templates themselves are categorized and they might add categories to the page they are used in, so do not confuse the two aspects! For example, the {{note}} template adds the page it is used upon to the Pages With Notes category. The note template itself is within the Editing Utility Templates category.
• As for templates, the license templates have been updated with new looks and easier to include in articles and pages. They have been categorized, and the feature to add pages they are used on to the corresponding category was added.
• The templates summary page was also updated so now it includes working samples of the templates and examples of their usage.
• License text was added for all the GNU License agreements and for the creative commons CC-BY license and the corresponding license template was updated to point to the kDemons page for the license text instead.

I have created a proposed categorization hive for kDemons wiki-pages and added categories for the kDemons namespace pages. You can review the current categorization hive here. If you have any ideas, suggestions or modifications, please feel free to change to the listing (please do not create categories yet however).

I’ve been working for a while on a new skin for the wiki feature of the site and I’m finally making progress. I’v einstalled the new skin and it is currently the default (and only) skin of the site. More modifications are to be made, but at this point, I reached a stable skin that is different from the standard one and still does not take a lot of resources. The new skin still keeps the layout of the standard MediaWiki interface (the skin is based on the MonoBook skin to be precise). It’s only small modifications but it sure did take some time and I hope that it will be a good start. I’ll post if any thing new happens regarding the interface of the site. That’s it for now. You can view the site in its new skin at the normal kDemons Wiki feature.

N.B. I think that from now on, I’ll just say “at the kDemons site” as all the actual content will be posted there.

New copyright templates were added and a templates’ summary page was created. As kDemons’ library of templates gets bigger, this page will be reorganized accordingly. The new templates created are: CC-BY, CC-BY-NC, CC-BY-ND, CC-BY-SA, CC-BY-NC-ND, CC-BY-NC-SA, GPL3, AGPL3 and LGPL3. 10:08, 14 June 2009 (UTC)

A new proposed structuring mechanism for the site’s contents was created. Here’s a brief overview and you can read more at kDemons:Patterns

What is “kDemons Patterns”

The kDemons Patterns project is considered to be one approach for avoiding duplication of content as much as possible without restricting the site’s content to a narrow/specific format. The project aims at creating different resources’ patterns that can be followed by welling contributors to add content to the site. In other words, we are trying to isolate/identify various distinct specific formats that can be used to represent information to the reader. After that, we will create a set of “best practices” for contributors to follow when creating resources in a certain pattern. To demonstrate the concept, let’s consider a simple example. One type of information that many editors might be interested in adding content within is the term definition type of resources. That is, resources that lists terms, abbreviations and commonly used acronyms along with their definitions. This is a good candidate to become a kDemons Pattern. The kDemons community can develop a friendly specification for such resources, specifying guidelines, best practices, and article formats that should be followed when adding such resources. We might agree to approach the subject using one of two ways: either writing articles in the main namespace for each term and then creating a glossary link page that links to these articles, or creating the specific terms articles all as sub-pages from the main page that includes the listing. After that, a group of developers might start working on adding content to this article pattern.

An initial setup of the kDemons Forums has been installed, configured and now, structured (still an incomplete draft). You can visit the forums here.

I tried to make the structure as simple as I could without making it impossible for a future user to find what he/she is looking for. Mainly, the forums contains a section called “kDemons Forums” (a little recursive, don’t you think) where it should be possible to discuss issues related to the kDemons site in general such as kDemons policies, categorization, features, software, interface and the like. The other sections are where the content will be. There’s a section for programming languages, .NET development. PHP development, and more is to be added soon. Hope this will be useful. Again, any help is appreciated!