SpearNZB

Version 1.1 of SpearNZB is mostly about databases and bug fixes. 

About a month ago TheTVDB was going up and down on a daily basis. More and more web sites and applications are relying on their great service and they are also in the process of moving to other technologies. 

Since this application relies so heavily on their database I started to think about how I decouple SpearNZB from them. If they decided one day to shutdown, or they revoked the key used to connect with their service, I would have allot of upset users. 

On top of this was the fact that the information is provided by the community and so if a volunteer doesn't add the show information, then it will not get downloaded. 

The answer seemed to be adding TV show information to the SpearNZB database. Currently this database gets information from both TheTVDB and TVRage. The idea is getting the best of both worlds. If there is another source to draw on in the future that can also be done without changing the application. 

This was quite a bit of work, and by using the application you shouldn't even know that it's happening. 

What's in the database at the moment are current series that are airing in the United States. If someone searches for a Series that is not yet in the database then it is added at the same time that it is added locally. 

Information is synced between TVRage and TheTVDB once a day. 

This also means that the load of SpearNZB on TheTVDB should be much lighter. Instead of every user synching every time they check for episode, it should only happen once a day on our server. 

I also want to encourage people to update TheTVDB, if you know that series that you are concerned about isn't getting updated on TheTVDB. Even though SpearNZB should still download, your media center won't update if it uses TheTVDB.

Java 6 Update 18

This version of Java has some changes that breaks Show Settings during downloading, and possibly other features. 

Windows Users might have noticed this, Update 18 is not available on the Mac currently. 

Bug Fixes / Changes

I can't say enough nice things about the users of SpearNZB. Everyone has been helpful in making it a successful project. 

Since the release of version 1 the question comes to mind on where to go from here. 

Some ideas. 

1. IPhone application. Allowing scheduling, RSS ext from the IPhone.

2. Integration into ITunes.

3. Movie search ability, similar to how Episode scheduling works. 

4. Add TVRage scraper. 

5. Online web page scheduling. Allow RSS selection and selecting an NZB file from a remote location. 

6. Show episodes downloaded in a media center kind of view and allow watching. 

Comment on these features or ask for your own. 

Thanks to all those who have purchased SpearNZB!!!! 

I hope that you find it useful, and look forward to enhancing it, making it the best NZB binary news reader on the market.

I'm constantly  blown away by the nice constructive feedback, that helps make this a better application. 

On that note there is a new version. Do I plan on doing a version every week? Not necessarily but I've made some improvements that I would like to get out there. 

1. No longer uses the working directory when downing the temp segment files. They are placed into the temp directory, decoded and placed in the working directory. This is done to avoid Spotlight from indexing these files.

2. Performance has been a thorn in my side, but it is now even better. Turn out it is not the downloading but updating the user interface that eats up much more of the CPU. 

Thanks to the registered :) user who found that with large (100's) of files in a NZB file that SpearNZB would grind to a halt. 

You might not have seen this, but if you have you should notice the number of files not affecting performance. 

You also can expect better performance with all downloads.

3. Put logging back in. Logging is either on or off. It is recommended leaving it off unless you are experiencing an issues that you would like addressed. Also this is not the same as the logs under the open menu. This is more for debugging purposes, that  generally you shouldn't be concerned with.

4. NZBMatrix now looking for both episode formats. Usually episodes on NZBMatrix are in the format s00e00. It was noticed with Top Gear the format ##x## was used. Now it searches for both, using the first as the primary. 

5. Show Monitor option placed in the Settings Dialog. Sorry for the confusion that most people were not able to find it, in the status bar. 

6.Fixed bug with problem files being re-downloaded from the watch directory. If a file could not be downloaded and repaired, it was downloaded again from the NZB Watch directory. 

Just a minor update today.

I noticed that on very large NZB files that there could be a very large slow down to where SpearNZB become unresponsive. 

This was not in the application code but in a build config setting. After changing this setting, I'm getting the same CPU usage (30-50% on 1 core).

This could have been the bug where over 100% CPU usage was reported. If so then hopefully it's the end of that problem. 

Happy Downloading,

Kraig

A few people have pointed out that the performance of SpearNZB could be better and I agree. I've noticed that downloading takes twice the CPU as say another product 'Unison'. I've spent allot of time trying different things, investing in profilers and the bottle neck seems to be in the Java runtime.

The reason why this seems to be so, is that the code that takes 95% of the CPU during download is the line to read from the network stream. There are different ways that this can be done, buffer sizes ext and it doesn't seem to make a difference. 

My experience however is that this doesn't cause myself as a user of SpearNZB much if any inconvenience. 

My media computer is a three year old PC running Vista with 2 Gigs of RAM, so not top of the line by any means. It is very common to be downloading at full speed (8 to 20 megabit) and watch a High res TV Show or movie via XBMC without noticeable side affects (Jerky Video, pauses ext...) Even during unraring/copying. 

If you can watch a high res video and download at 8 Megabit without any issues, then it would seem that you could surf the web, do email at the same time or whatever else. 

Now to be fair, the faster the download the more resources will be used, so at 50 Megabit this might not be true. I don't know because I can't test that. 

The screen shot below shows typical resource usage on my development machine (IMac 2.4 Intel Cure 2 Duo, 2 GB RAM)

At the beginning of a download CPU usage can go up to 120% (50% per CPU) and then stable out at around 30%. It does jump up to 50% depending on what it's doing, but 30-40 should be the average. 

Hopefully for most people this is acceptable. If not check in every so often, you never know....

Most of the time SpearNZB can find episodes for your series without any problems. However there are certain series titles that can cause a problem. 

How the search works is that SpearNZB queries RSS feeds or uses its internal database for the title of the series, and a season episode number.

This is how you might look for something on a Web Site like Newzbin. You would time something like CSI s09e01 to find CSI season 9 episode 1. 

Where we run into problems is the actual show is CSI: Crime Scene Investigation. Most of the time the release will just have CSI. Or if it's a spin-off it will have the entire series name. 

The trick is that for a few special cases, like this one you need to tweak the series name to matches for the file names that get posted. 

This also works differenty depending on what you use to search episodes (Newzbin, NZBMatrix, TVNZB or SpearNZB)

For NZBMatrix the title needs to be set to what we expect to find in a search. So we rename to CSI

Now SpearNZB will search for CSI and the season episode number. Also don't forget to set the category for NZBMatrix to get the resolution that you desire.

TVNZB and SpearNZB-Database works by giving you a list of all of the names that make up a series. 

Highlight the series and press settings.

In this case we want to remove everything but CSI and press the add button to add x264. So when SpearNZB goes to search for this episode it will find files that have 

CSI s##e## and x264. 

We could also add more text to narrow down a search if that was needed as well.

With Newzbin you first need to create an RSS Feed that will return valid items to search. We would create an RSS feed for CSI x264 in the TV Section. 

Here we actually want the title CSI: Crime Scene Investigation because on Newzbin they have wonderful moderators that use proper names.

90% of time you don't have to worry about this, but if your not finding episodes for your favorite shows this should help in finding an answer. 

Happy Watching.....

Beta 2 was by far the most successful beta. A lot of issues were found that I never would have found on my own including 

1. Enhanced SSL certificates. Thank you NewsDemon

2. Category not getting correctly set with Newzdemon

3. Spotlight trying to search the contents of the download thus killing performance. 

Some new features, most notability the NZB database is back, not on your system but in the cloud somewhere. 

This means your system is not downloading and indexing the NZB entries, that is done elsewhere. 

You can also use this database for episode searches, so you don't need the RSS feeds. 

Keep in mind however that RSS feeds are still a great way to get NZB files and they are usually reviewed by members of the web sites. There is no review in the SpearNZB database at this time, but I've personally haven't had any problems.

There will be more videos and instructions in the days to come, but I had to get this out due to the current version timing out on Oct 1st so no time at the moment. 

Enjoy...

Kraig

My original intent with SpearNZB was to have a desktop application that didn't rely on any external web sites. Allot of progress was made to that goal but I decided that for now I should pull out that feature. 

Configuration, bandwidth, speed and disk space was the main reasons that I removed it. 

If the Websites that offer the RSS feeds go away then rest assured it will be back. For now RSS feeds are just much more efficient.

Instead I've added much better support for RSS feeds, including Newzbin, NZBMatrix and TVNZB. 

Compression for Astraweb was also added, so I'm the second Usenet client to implement it. Good for me.

TV Shows can be auto-named. I didn't think that this would be all that useful, but after using it for awhile, it's nice to have consisent names with the Series, episode and episode title.

Season can be placed in their own directories for even more organization.

Linux build is also back for you true geeks. 

When the beta went out a couple of months ago what seemed to be the feature that most people were excited about downloading TV Series and episodes. 

The biggest issues were that it took about a day to build the private NZB database from Usenet and the setup for selecting a TV Series and episodes was both problematic and confusing. 

The reason for the confusion was trying to make it more general like being able to select any newsgroup to look for episodes, but then a user would need to know what newsgroup to select. 

Well those dark days are over. First the newsgroups that are used for used for searching are downloaded from the SpearNZB web site. This means that the initial day or more to setup the database should take a few minutes. 

Also integration with TheTVDB makes episode so much cleaner. 

You simply add a series and select the episodes you would like to download. If an episode has not aired yet, then it will be downloaded after it does air.

If your a previous beta user, you should delete the database in your <UserDir>/newdb directory. 

The database will be built again fairly fast, but you will have to re-enter your RSS feeds. (Sorry)

I'll be posting better instructions and How To Videos as time permits.

Kraig....