10:40 a.m. Horizon is back up and in working
order. The problem has been resolved. If you experience any
additional problems, please call us right away. Thank you for your
patience! IHLS - Decatur Office.
10:15 a.m. We are experiencing a problem with
our transaction logs filling up in Horizon. Once full, we cannot
do any transactions (CKI, CKO, etc.). We are working with John
Craig to resolve the problem, and he is estimating us being back up in
15 to 30 minutes. As soon as we have any additional information,
we will post it to the elCAT listserv and this page. Thank you for
8:06 a.m. It appears the problem we had last
week regarding e-mail has been resolved. You can resume sending
e-mail notices out. If you experience any problems with e-mail
notification, please call us here at the Decatur office. Thanks
you. IHLS - Decatur office.
4:41 p.m. The issue we had with our internet
service here in-house has been resolved. Unfortunately, we are now
experiencing an email issue and are unable to follow up on it until
Tuesday with our IT department. We are recommending you do not
send out e-mail notices until we have had a chance to work with IT
Tuesday morning. We apologize for any inconvenience this may
cause. Thank you for your patience! IHLS - Decatur office
8:47 a.m. Searching in HIP should be working
correctly now. Please let us know if you continue to experience
any problems. Thank you for your patience, IHLS.
8:40 a.m. We are experiencing trouble with
searching in HIP. The problem is being worked on and we hope to
have it corrected soon. Thank you, IHLS.
6:57 p.m. The problem with inaccurate searching
in HIP (an indexing issue) has been resolved. Thanks again for
your patience. IHLS
4:45 p.m. There is a problem with searching in
HIP. Searching in the Staff Client does not seem to be affected.
We are working with AlphaG-Consulting to resolve. Best estimate at
this time is approximately 2 hours. We will update this page and
the LLSAP listserv once resolved. Thank you for your patience.
IL Heartland Library System.
8:11 a.m. HIP is back up. Thank you again
for your patience. RPLS
7:06 a.m. It appears HIP is down, and is
causing middle tier errors. While down, the request system will
not work, although you can perform other tasks in Horizon. We are
working to resolve the problem and will update the system status page as soon as
we have something to report. Thank you for your patience. RPLS
6:45 p.m. Reindexing should be complete by 9 p.m. (or earlier)
When the rebuild is complete, HIP will go down for a few minutes while
the new indexes are switched in, and all should be back to normal. Thank
||4:00 p.m. HIP indexes are having problems; searches
were returning blank lines. We are working on getting it fixed, but HIP
will have to be reindexed. The backup indexes are in place and functioning, but
they are out of date -- new items added since the upgrade will probably not
appear. Thank you for
your patience. RPLS
||10:19 a.m. ICN made some adjustments and we seem to
be running much better. If anyone continues to have trouble with
response time, be sure to call. Thank you again for your patience.
||9:27 a.m. We are experiencing extremely slow
response time. We are working with ICN to resolve. We will
update this page as soon as we have more to report. Thank you for your
||2:01 p.m. The problem with HIP seems to have been
resolved. Please let us know if you are still experiencing any
problems with HIP or Horizon. Thank you, RPLS
||We are experiencing a problem with
HIP, which also creates connection problems with the Horizon database.
We are working to resolve and will update this page as soon as we can.
Thank you, RPLS.
||Problem was resolved once HIP and
the Horizon databases were rebooted. Thank you for your
||9:40 a.m. Horizon is up and
running, but there seems to be a networking issue affecting many of
our libraries. We are working with ICN to resolve. We will
update this system status page and the LLSAP listserv as soon as we
know something more specific. Thank you for your patience.
||9:50 a.m. We are back up and working. Sorry for the
inconvenience. Thank you. RPLS.
||12:29 p.m. Good afternoon, LLSAP
HIP is back up. We will be making
some changes in the background to try to ensure that it stays that way!
Please log out and log back in to Horizon, so you will
be able to place requests. You should see no middle tier errors.
If there are problems, please let us know.
Thank you for your patience.
||9:28 a.m. Many libraries are logging in for the
first time today and getting a "middle tier" error. That error is due
to the problem we are having with the HIP server, which is currently down.
SirsiDynix and Jo are working to resolve. In the meantime, you may
continue with logging into Horizon by simply answering OK to the error.
You can perform all tasks in Horizon with the exception of placing requests.
Once the HIP server is back up, you will then be able to place requests.
We will update this system status page and the LLSAP listserv as soon as we
have something to report. Thank you, RPLS.
||8:43 a.m. We are experiencing problems with the
HIP server. Jo is working with SirsiDynix to resolve. We will
update this page and the LLSAP listserv as soon as we have something to
report. Thank you for your patience! RPLS
||5:54 p.m. Update #7 -
Good afternoon, LLSAP libraries --
We are back from Champaign and are tired but happy.
The upgrade is complete, and with a few glitches, everything seems to be
running smoothly. We expect a couple of bumps in the road from time to time,
but we'll get them worked out.
I want to publicly thank the people who made it
happen: Terry and Sue (can't ever thank them enough); John Craig of Alpha-G
Consulting (who continues as our server maintenance guru); John Gillespie,
Alison Felt, Peter Fripp, and Jennifer Bills of SirsiDynix; Rich Mann of
Illinois Century Network (who worked his network magic, as usual, and
provided TONS of help getting the servers set up in Champaign); Steve Look,
who got our new switch and power strips in record time; Matt Orr of PC
Discounters (the chief hardware supplier, who went above and beyond the
call); the RPLS staff who backed us up and tested our procedures -- Angela,
Doris, Louise, Rachel, and Bev; and most of all you, our member libraries,
who partnered with us, supported us, and ran your libraries for three days
without a catalog or a circulation system.
We will be leaving the building now, but Sue has
the emergency cell phone and Terry and I are less than 10 minutes away if we
need to come back in. Otherwise, we'll start in again tomorrow at 8AM and
see what else needs to be done. We are running DayEnd tonight, which, if it
works, will generate notices for tomorrow morning. We'll let you know if
there are any issues.
5:45 and all is well. Signing off from Upgrade
Thank you again.
||Update #6 -
Horizon is back up. Please check your message
from the LLSAP listserv for further directions.
||7:00 p.m. Update
It's been a busy afternoon!! We have made a great deal of progress.
The database is in place. We are running backups, cleanup, and various housekeeping tasks. Several loose ends still to fix, but it's looking good.
HIP is working. Requests work. Most of our customizations are back in place. Patron position in the hold queue does NOT appear.
Our plan is to finish the cleanup in the morning, do an orderly shutdown of everything, and pack the servers to move to Champaign. We plan to be there at about 1:00 or so. It is going to take a few hours after we arrive to get everything moved in, installed, hooked up, started, and tested. We still plan on being back up by the end of the day (schools, this is probably past the end of your day, so check System Status first thing Friday morning to see what happened).
We're having a few issues with Sybase and the new client installation out in the libraries. So far, we've been able to get them all resolved. If you have trouble when we come back up, please call and we'll get you up and running as fast as we can. There may be a bit of a line-up for tech support when things first start up again, but please be patient and we will get to all of you.
Debt Collect should be working. There is still one SIP issue to resolve, but SirsiDynix is aware of it and will continue working with the affected library until it is fixed.
Thank you all very much for your cooperation in this major venture!
||12:25 p.m. Update
The upgrade is going very
well. Horizon and HIP are upgraded, but they are still a work in progress.
Please don't try to access Horizon
yet. It's still a major construction zone, and it can't be used in any way.
You may load software on your PCs, but you won't be able to connect.
HIP will connect, but again, it can't
be used. If your patrons ask, say please no requests or searches until
Thursday (or perhaps Friday, if you want to ease into the process -- your
Both Horizon and HIP will be up and
down repeatedly until all the construction is finished. We will take
everything totally down again tomorrow for the move to Champaign.
Thank you for your patience.
||9:15 a.m. Update #3 -
SirsiDynix has just started to install the new version of Horizon on the server. All the data has been copied over from the old server to the new, and SirsiDynix is making a backup copy on our new backup array (a bunch of hard drives on a separate server). After some testing and configuration, the new Horizon will eventually meet up with the existing data. There will be some work involved to make sure all the data goes into the right place, and to get some of our specialized features to work again.
But we are making progress! We are reasonably on schedule -- still on track to come up late tomorrow (Thursday).
As far as we can tell, most of you are doing well with installing the new software on the staff PCs in your libraries that run Horizon. We would encourage you, if you have not tried to do the installation, to do it today if possible. If you have problems with the installation, it might be easier for us to work with you now, rather than when the database first comes back up. Please call us if you have questions or difficulties.
||3:00 p.m. Update #2 -
All the framework has been set for the
database on the new server.
We're ready to start the first steps
of moving the data to its new home. This is a long process and will take the
rest of the day (and into the night, most likely). When it is finished,
SirsiDynix will perform some cleanup and testing. After all the data is
moved and tested, the process of upgrading the database will begin.
The HIP team has been working, too, so
HIP will be ready when Horizon comes back up. They are almost finished for
the day, because they are now waiting for the database to be copied, in
order to do the next stage of building HIP.
This is probably the last update for
today, as nothing else of significance will be happening. We have many tasks
to do while the data is copying, but everything is behind-the-scenes work.
Update #1 -
SirsiDynix is working on
Sybase on the new servers. After that work is complete, they will load
Lincoln Library's comres (NewsDex) database. If that goes well, they will
begin copying the main Horizon database to the new server. With the amount
of data we have, that's going to take a considerable amount of time.
At the same time, another SirsiDynix
technician is working on HIP (for Horizon and the comres database). The SIP/DebtCollect
folks at SirsiDynix are contacting people in the libraries that use those
services. We have a lot of multitasking going on.
We will continue to update you as
things progress. Thank you, RPLS
| 3/15/11 - 3/17/11
upgrade/migration/co-location in progress. Please do not attempt to
access the Horizon database until notified. Thank you, RPLS.