Many people have reported Fusion repeater lock-ups that required the repeater to be shut down (power removed) and rebooted. I’ve had repeater lock happen as well with an HRI-200 directly connected to the DR2X. In this particular case I believe the event was triggered by sending a number of commands from the radio (such as the GM mode) which resulted in a race condition within the firmware.
I have previously observed that invalid data (I.e., with a lot of bit errors or badly formed packets from a hotspot) can also cause the lock-up. My assumption is that this is a problem of not validating inputs to a function prior to executing the function. The invalid data is not handled correctly and causes one of the lock-up faults I mention below to occur.
I have observed that data is more likely to be invalid if there is a lot of noise on the signal. Also a major source of problems are badly behaving hot spots that inject all sorts of crap into the data stream. There’s no control over these devices and it just gets worse as there is more work being done converting from one thing to another thing, etc.
Communications software is EXTREMELY HARD to write and much of what I see out there is poorly documented (which leads to bad code) and poorly tested.
Reflectors such as YSF or FCS are pretty dumb when they relay signals. They look for an incoming packet from one of the connected hotspots and just send that same packet out to all the other hotspots (including bridges to WiRES-X). Garbage in, garbage out. To address this I have modified the YSF reflector software to be very aggressive at dropping packets with CRC errors, invalid data in fields, and most importantly invalid FICH headers. The FICH header is extremely important for everything to work right. (You can look up FICH headers in the System Fusion documentation published by Yaesu.) This modified software is running on MNWis, Princeton Kentucky, MVARC (Idaho) and America Link. It has dramatically reduced the number of problems with WiRES-X problems.
So what is a “lock-up”. A lock-up occurs when the processor looses track of what it was doing and either halts or ends up executing invalid code. In either case it is no longer processing the code it was intended to run. Events that cause the processor to, using a technical term, go out to out to lunch can result of errors in the software that cause it to happen, corrupted memory, invalid states in a state machine, incorrectly handled race conditions, or a really nasty thing called priority inversion (where a high priority task which is pre-empting use of the CPU is waiting for a low priority task to complete, which can’t complete because, well, it’s not high priority.)
How to solve the problem?
First of all, the repeater firmware should never crash. Since it is very hard (but not impossible) to write code with no bugs, most continuously operating real-time systems will incorporate a “watch dog timer”. It is the purpose of this timer to detect when the “wheels have fallen off the processor” and perform a reset.
Since we can’t modify the repeater’s software, there are some actions we can take.
First would be to prevent bad data from getting into the system. A careful examination of the logs possibly combined with audio recordings could be helpful.
A quick fix might be to install a timer on the power supply such that the repeater’s power is removed for a short time once or several times a day.
I have a technical concept for a generic lock-up detector that would reboot the repeater in the event of a lock-up without the need to make any modifications. Not sure if I’ll work on this because I don’t know how much interest there would be. This solution would also work with non-Yaesu repeaters.
One further note. I made an observation that this problem was more likely to occur on 2 meters than on 440. Why? My theory is that there is a lot more interaction with the environment on 2 meters. Computer networks, small power supplies, computers, almost everything digital generally creates a lot more noise on 2 meters than it does on 440 MHz. I observed a situation where mixing products were occurring on a 2 meter repeater in a noisy environment. This repeater would lock up regularly. My theory was that the mixing products contained a lot of noise along with the C4FM signal causing a high number of bit errors and thus crashing the repeater. This was observed on a DR1X.
If you are struggling with this problem, I suggest you contact Juan at Yaesu Customer Service with your details. Provide Juan with as much information as you can.
Running two HRI-200’s on one IP Address
I use VyprVPN from GoldenFrog. It is necessary to turn of NAT for your account. That means that each VPN connection is a raw connection straight to the Internet. YOU HAVE NO PROTECTION OTHER THAN WHAT YOU PROVIDE!!! (The same company will rent you a VPN that you can host on your own cloud provider where you can control port forwarding – but I suspect this is well beyond most folks.)
What Does the HRI-200 Actually Do?
What’s the difference between using an HRI-200 and directly connecting to the radio?
1. It is the only option for connecting a repeater to WiRES-X.
2. It uses UDP communications sent directly to the Room. So network communications are very efficient.
PDN uses TCP instead of UDP. (Why use UDP? UDP is simple. If anything gets lost, it is dropped. Good for real-time communications. If a packet is missed with TCP, it will keep retrying up to a limit which is not good for real-time. VoIP, VPN’s, Netflix, etc. all use UDP.)
The PDN TCP packets (non-HRI-200) are sent to a server in Japan. This server converts the packets to UDP and then sends them to the selected room in the same way the HRI-200 does. So by using the HRI-200 you bypass a drip to Japan and a potentially problematic conversion from TCP to UDP and back to TCP for the return trip to your station.
Yaesu uses the TCP approach to get around forwarding of ports. The WiRES-X PDN software establishes a TCP connection with the Yaesu server. It’s through that connection stations can contact you even though you don’t have an open port. With the HRI-200, stations just get your IP address and send you packets directly. This is why PDN cannot efficiently host a room – all room traffic would have to go to Japan for the conversion process.
In a previous post (2016) I described what the HRI-200 does. Why it is the way it is is somewhat historical from the history of WiRES, WiRES-II, and now WiRES-X. Obviously an Ethernet port on the back with an embedded processor to run things would be a better way to go. I can only guess that the ROI is simply not there for Yaesu. They do, after all, have to make a profit.
One further note. It’s probably a good thing non-Yaesu equipment doesn’t connect to WiRES-X. The biggest problem in bridging hotspots (YSF, FCS, etc.) to WiRES-X is that the hotspots are bad neighbors. They generate bad packets, do crazy things, and can really trash a network. As a room operator we have no control over who tries to connect, so at least the Yaesu equipment has gone through compatibility testing. Believe me, very little testing is done on hotspot software.
MNWis (21493) bridges to YSF (21493) via a YSF Reflector where I have extensively modified the code to drop bad packets, control information, and all manner of things that shouldn’t see there way into the WiRES-X node. I had to do this because we were constantly having hotspots connect and mess everything up. It got very, very tiring working with the (mostly new) hotspot owners to fix their problems. I mean the conversations were endless and constant! So I just dropped anything that looked bad in the YSF Reflector software and the problems went away as well as a few hotspots who could no longer connect or talk to people.
System engineering is critical. We need Yaesu to play the roll of system engineering to keep all the Fusion products working with each other. It’s so nice that all the Fusion equipment basically works the same way and does the same thing all of the time!
BTW, you can bring your questions/comments to the MNWis Fusion Technical Net held Monday’s at 7:30 PM Central in the above mentioned locations.
When a WiRES-X node keeps going online then offline, this type of problem is almost always the result of a networking issue. A networking problem will occur and will wack out the HRI-200 software. Once the software has been wacked out, it will continue to cycle in and out of whatever room it is connected to. Restarting the WiRES-X software usually solves the problem.
1. Turn off UPnP. It should never be used!!!!!!
2. Turn off all quality of service features like those to improve Xbox gaming or VoIP functions. (Yes I know we’re using VoIP so you’d think it would help.)
3. In fact turn off everything you don’t have to have.
4. Some routers are total crap and need to be rebooted every few weeks.
5. You’re not using Wi-Fi are you? Wired is always better than wireless – we should know! Drop-outs in Wi-Fi can cause this issue.
1. Something as simple as the ISP dropping your connection for a few minutes in the middle of the night when they’re doing maintenance.
2. The ISP just doing a bad job of getting packets to you during prime evening Netflix viewing.
3. The ISP switching your IP address. (I think CenturyLink just does this for the fun of it.)
1. Make sure Win10 doesn’t put the USB interface to the HRI-200 to sleep. It really, really likes to do this. Just going to the obvious place in the control panel to turn this feature off doesn’t really turn it off. See the documents on running Win 7/10 24x7x365 remotely in the Fusion Help section at HamOperator.com.
2. Make sure the computer isn’t crap. Cheap computers may not have quite the reliable communication interfaces that we need.
3. Banish all RFI and ground loops. Make sure RF is not getting into the USB interface to the HRI-200. This can cause message errors and really screw things up.
4. Make sure you computer actually has enough power to run the HRI-200 reliably. Get one of those USB power thingies and make sure you have a solid 5VDC under load.
Over the air:
1. Bad data on the WiRES-X network can mess things up. This happens mostly with hotspots that are bridged into the WiRES-X room. On MNWis and a few other networks we’ve banished this problem by banishing FCS and running YSF server software that banishes bad data from hotspots. So if you have this problem in one room, say AmericaLink and not others, there may be nothing you can do.
2. Yaesu radios that are running really old firmware or have not had the firmware updated correctly (I.e., “I did the main CPU but I’ll get around to the DSP later.”)
The port check should not be relied upon to definitely prove it’s working or not working. The test is not exactly the same as actually communicating with the Yaesu list servers and the room server. So it can lie to you with false positives and false negatives.
You may also experience a similar problem if the Yaesu list servers are not able to keep up with the demand or if maintenance is being done. (When they do maintenance in the wee hours of the morning, that’s right in the middle of the day for us.)
And keep in mind the MNWis Fusion Technical Net held in MNWis WiRES-X and YSF # 21,493 Monday nights at 7:30 PM Central.
WiRES-X and USB Problems
If you’re having problems with USB problems when using WiRES-X with or without other programs, read on.
The WiRES-X software is pretty good at identifying the HRI-200 vs. connecting directly to the radio. Windows, however, is very bad at managing USB ports. The WiRES-X software makes two USB connections: One as a virtual COM port and the other as a sound device. Windows is also very bad at managing sound devices.
Virtual COM ports can be identified by the port number (a real pain as it is always changing – see below) or by identifying the hardware via the manufacturer and product number that is burned into the USB device. Generally software that does not provide an option to specify the COM port (as with the HRI-200) is using the hardware identification to establish the association.
When Windows sees a USB device it will enumerate it and create a driver for it. If you move it to another port it will, for reasons we can only guess at, create another driver. It will keep making new drivers as long as you have USB ports and USB devices left. Why, we can only guess. Periodically it may be necessary to go through a clean out all of these “drivers” that Windows keeps making and saving until the end of time. You can do this from the command line or from within Device Manager.
I have found WiRES-X to play nicely with other devices. For example I have no problem running hotspots, Ham Operator Deluxe, WSJT-X, etc. on the same computer that is running WiRES-X.
You may find it beneficial to use a USB 2.0 vs. 3.0 port. Why? USB limits the number of connections that can made per USB controller. Keep in mind that several of the ports on you computer may use the same controller. Also, each USB device may use more than one connection. As I recall sound devices generally use two connections: One for the audio and one for control. A USB 2.0 controller supports 255 connections. How do I know? I actually ran up against this limit using a couple of 16 port USB hubs. USB 3.0, while faster, does not permit as many connections. I don’t remember the exact number as it has been some time, but the number SIX sticks in my brain. As you might imagine, it is very easy to burn up six connections. Also keep in mind that the USB controller may be servicing hardware that is internal to the computer. It all depends on how the computer was designed. Top end computers will generally have a higher controller to port ratio. Bottom line: you may want to use USB 2.0 if you have been using USB 3.0.
Perhaps try something like this:
1. Blow away the WiRES-X and any conflicting software. Make sure everything is unplugged from USB.
2. Blow away all of the USB drivers. Just uninstall them. Don’t worry, the files to recompile a driver are still on the computer. [Device Manager with admin. Enable option to view all devices. Delete drivers under COM ports. Delete drivers under sound devices (except for hardware built into your computer.)]
3. Install the WiRES-X software. Allow it to associate with the USB devices it needs by plugging in the HRI-200, etc. If you’re not using an HRI-200, you may find it beneficial to use an external USB audio device for WiRES-X. If you can use USB 2.0 ports.
4. With WiRES-X happy running, install the other software. Plug in the necessary USB devices so it can similarly make the association. At this point everybody should be happy with both programs running just fine.
5. Write down which device was plugged into which USB port. In the future, always use the same USB port for the devices. We don’t want Windows creating more drivers.
Good luck solving the USB problems!
YSF WiRES-X Unintended Node Switching
A problem exits with YSF/MMDVM hotspots bridging to WiRES-X rooms. Here’s what happens.
A user has a hotspot and accesses it using a Fusion HT. They use the WiRES-X control mode to change the hotspot to different YSF Reflectors. Let’s say the user wants to access MNWis (Room #21,493):
- They enter the number of the YSF Reflector, “US MNWis 21493”, which is 37,624. The hotspot switches to US MNWis 21493 which is bridged to WiRES-X Room # 21,493.
- The user leaves their radio in the WiRES-X control mode. They hear stations, but when they talk, nobody answers and nobody is talking.
Here’s what happened: When they keyed up to transmit, their radio sent a command to switch to Room #37,624. The Wires-X node that was previously connected to #21,493 now connects to #37,624 which belongs to a very nice gentleman in France.
How does this happen?
The Fusion transmitted data includes callsign fields CSD1, CSD2, and others. We’re interested in CSD1. This 10-byte field contains two items: The room number of the connection and; The TxID or DP-ID of the radio. A typical CSD1 looks like 21493F0yxh where 21493 is the room number and F0yxh is the DP-ID. The ‘21493’ in this field will cause a Wires-X node to switch to that room.
When the radio IS NOT in Wires-X control mode, the CSD1 field will be: ‘*****F0yxh’ and no switch will be made.
DON’T USE WiRES-X control mode to talk through a hotspot or MMDVM!
Workarounds and fixes: (Updated 23-Aug-2019)
Node operators who are experiencing this problem should use the WiRES-X block feature to prevent the node from switching to the room that has the YSF reflector number.
For example: Assume that our node connects to MNWis, Room # 21,493. Our node also provides bridging to “YSF MNWis 21493” which is # 37,624.
- In the View->Node-Info(N) window press “Add”.
- In the “Input ID” dialog box enter the YSF reflector number. In our example this is “37624”.
- Press “OK” then “Close”.
- When a station using the YSF bridge still has WiRES-X control mode enabled accesses the node, the command to switch rooms will be rejected by the WiRES-X software. The software will indicate a rejected attempt to switch rooms.
As an alternative, set the node so that connection changes are not allowed.
- File->Settings->Call Settings->Uncheck “Round Room Connection”.
- Un check “Accept calls while in Round Room QSO.
- Check “Return to Room”
- Fill in the WiRES-X room number. Example: 21493.
YSF server side solutions:
I maintain a version of YSF Reflector that has filters to prevent hotspots causing problems with WiRES-X nodes. My first step is to drop all packets that contain an incorrect WiREX-X room number. Eventually I plan to replace the first 5 bytes of CSD1 with ‘*****’. This last step is difficult to do because it requires the reflector to decode the data, modify it, then recompute the CRC. This also involves working with the interleaving and forward error correction. It is obviously easier to just drop the packets, but that may confuse people on the YSF side since nobody on the WiRES-X side will hear them.
If you are interested in running the enhanced YSF Reflector software that will fix this problem at some point, please contact me privately.
Update: A corrected version of WXScheduler has been uploaded (15-Mar-2021)
Yaesu does not provide a mechanism that allows the WiRES-X software to be controlled by another program, i.e., having another program switch to a certain Room when a net starts.
Windows does, however, permit another program to send events to a program. Each window, menu item, and dialog in a Windows program has a unique identifier. It is possible to use these identifiers to send “message” to the WiRES-X software.
The WiRES-X Automation Project’s purpose is to bring together people who are interested in developing this technology and sharing their results.
To get things going, here are two mechanisms for automating WiRES-X:
1. AutoIT: http://www.autoitscript.com
2. Python – an excellent programming language found at python.org
Dave, N9TOW, provided the following information:
Packages I have installed on my WiresX system.
pip install -U pywinauto
Script that executes changing channels on WiresX app
from pywinauto import Application
app = Application().connect(path=”C:\Program Files (x86)\YAESUMUSEN\WIRES-X\wires-X.exe”)
dialogs = app.windows()
Bill, W9LBR, developed a Python program that runs on the WiRES-X computer. This is from his description:
WXscheduler.pyw is a Python 3 program that automates certain aspects of
Wires-X Room/Node connections according to a userrdefined schedule.
To that end WXscheduler.pyw must be run on the same Windows PC
that hosts the Wires-X application controlling either an HRI-200 or
a USB connected radio that supports PDN operation.
– Restart Wires-X App
Manual operation: Button that causes a Force Disconnect
Bonus operation: Displays Last Heard information that the Wires-X app
updates once a minute. Converting Lat/Lon into Grid Square if available.
A zip file containing instructions and the Python program is available at WXScheduler. (Corrected 15-Mar-2021)
Mobile WiRES-X Nodes
FT2D, FTM-100, and FTM-400 Can Now Be Used As Mobile Node
Yaesu has now released software and firmware that enables the FT2D, FTM-100 and FTM-400 to be used as a WiRES-X mobile node. You’ll need to update the radio’s firmware and install WiRES-X software version 1.510 from HRI-200 and radio downloads at Yaesu.com. Instructions are available at the download site.
A document listing the current firmware and software for Fusion rados is on the Fusion Help page here.
Stop by for the MNWis Monday night Fusion Technical Net for more information.